Thread: Declarative partitioning - another take

Hi,

Attached is the latest set of patches to implement declarative
partitioning.  There is already a commitfest entry for the same:
https://commitfest.postgresql.org/10/611/

The old discussion is here:
http://www.postgresql.org/message-id/55D3093C.5010800@lab.ntt.co.jp/

Attached patches are described below:

0001-Catalog-and-DDL-for-partition-key.patch
0002-psql-and-pg_dump-support-for-partitioned-tables.patch

These patches create the infrastructure and DDL for partitioned
tables.

In addition to a catalog for storing the partition key information, this
adds a new relkind to pg_class.h. PARTITION BY clause is added to CREATE
TABLE. Tables so created are RELKIND_PARTITIONED_REL relations which are
to be special in a number of ways, especially with regard to their
interactions with regular table inheritance features.

PARTITION BY RANGE ({ column_name | ( expression ) } [ opclass ] [, ...])
PARTITION BY LIST ({ column_name | ( expression ) } [ opclass ])


0003-Catalog-and-DDL-for-partition-bounds.patch
0004-psql-and-pg_dump-support-for-partitions.patch

These patches create the infrastructure and DDL for partitions.

Parent-child relationships of a partitioned table and its partitions are
managed behind-the-scenes with inheritance.  That means there is a
pg_inherits entry and attributes, constraints, etc. are marked with
inheritance related information appropriately.  However this case differs
from a regular inheritance relationship in a number of ways.  While the
regular inheritance imposes certain restrictions on what elements a
child's schema is allowed to contain (both at creation time and
after-the-fact), the partitioning related code imposes further
restrictions.  For example, while regular inheritance allows a child to
contain its own columns, the partitioning code disallows that.  Stuff like
NO INHERIT marking on check constraints, ONLY are ignored by the the
partitioning code.

Partition DDL includes both a way to create new partition and "attach" an
existing table as a partition of parent partitioned table.  Attempt to
drop a partition using DROP TABLE causes an error. Instead a partition
needs first to be "detached" from parent partitioned table.  On the other
hand, dropping the parent drops all the partitions if CASCADE is specified.

CREATE TABLE partition_name
    PARTITION OF parent_table [ (
  { column_name WITH OPTIONS [ column_constraint [ ... ] ]
    | table_constraint }
    [, ... ]
) ] partition_bound_spec
[ PARTITION BY {RANGE | LIST} ( { column_name | ( expression ) } [ opclass
] [, ...] )

CREATE FOREIGN TABLE [ IF NOT EXISTS ] table_name
  PARTITION OF parent_table [ (
  { column_name WITH OPTIONS [ column_constraint [ ... ] ]
    | table_constraint }
    [, ... ]
) ] partition_bound_spec
  SERVER server_name
[ OPTIONS ( option 'value' [, ... ] ) ]

ALTER TABLE parent ATTACH PARTITION partition_name partition_bound_spec [
VALIDATE | NO VALIDATE ]

ALTER TABLE parent DETACH PARTITION partition_name

partition_bound_spec is:

FOR VALUES { list_spec | range_spec }

list_spec in FOR VALUES is:

IN ( expression [, ...] )

range_spec in FOR VALUES is:

START lower-bound [ INCLUSIVE | EXCLUSIVE ] END upper-bound [ INCLUSIVE |
EXCLUSIVE ]

where lower-bound and upper-bound are:

{ ( expression [, ...] ) | UNBOUNDED }

expression can be a string literal, a numeric literal or NULL.

Note that the one can specify PARTITION BY when creating a partition
itself. That is to allow creating multi-level partitioned tables.


0005-Teach-a-few-places-to-use-partition-check-constraint.patch

A partition's bound implicitly constrains the values that are allowed in
the partition key of its rows.  The same can be applied to partitions when
inserting data *directly* into them to make sure that only the correct
data is allowed in (if a tuple has been routed from the parent, that
becomes unnecessary). To that end, ExecConstraints() now includes the
above implicit check constraint in the list of constraints it enforces.

Further, to enable constraint based partition exclusion on partitioned
tables, the planner code includes in its list of constraints the above
implicitly defined constraints.  This arrangement is temporary however and
will be rendered unnecessary when we implement special data structures and
algorithms within the planner in future versions of this patch to use
partition metadata more effectively for partition exclusion.

Note that the "constraints" referred to above are not some on-disk
structures but those generated internally on-the-fly when requested by a
caller.

0006-Introduce-a-PartitionTreeNode-data-structure.patch
0007-Tuple-routing-for-partitioned-tables.patch

These patches enable routing of tuples inserted into a partitioned table
to one of its leaf partitions.  It applies to both COPY FROM and INSERT.
First of these patches introduces a data structure that provides a
convenient means for the tuple routing code to step down a partition tree
one level at a time.  The second one modifies copy.c and executor to
implement actual tuple routing.  When inserting into a partition, its row
constraints and triggers are applied.  Note that the partition's
constraints also include the constraints defined on the parent.  This
arrangements means however that the parent's triggers are not currently
applied.

Updates are handled like they are now for inheritance sets, however, if an
update makes a row change partition, an error will be thrown.

0008-Update-DDL-Partitioning-chapter.patch

This patch updates the partitioning section in the DDL chapter to reflect
the new methods made available for creating and managing partitioned table
and its partitions.  Especially considering that it is no longer necessary
to define CHECK constraints and triggers/rules manually for constraint
exclusion and tuple routing, respectively.

TODO (in short term):
* Add more regression tests and docs
* Add PartitionOptInfo and use it to perform partition pruning more
effectively (the added infrastructure should also help pairwise joins
patch proposed by Ashutosh Bapat [1])
* Fix internal representation of list partition bounds to be more efficient


Thanks,
Amit

[1]
https://www.postgresql.org/message-id/CAFjFpRfQ8GrQvzp3jA2wnLqrHmaXna-urjm_UY9BqXj%3DEaDTSA%40mail.gmail.com

On Wed, Aug 10, 2016 at 7:09 AM, Amit Langote
<Langote_Amit_f8@lab.ntt.co.jp> wrote:
> Attached is the latest set of patches to implement declarative
> partitioning.

Cool.  I would encourage you to give some thought to what is the least
committable subset of these patches, and think about whether it can be
reorganized to make that smaller.  Based on the descriptions, it
sounds to me like the least committable subset is probably all of 0001
- 0005 as one giant commit, and that's a lot of code.  Maybe there's
no real way to come up with something more compact than that, but it's
worth some thought.

> 0001-Catalog-and-DDL-for-partition-key.patch

+      <entry><link
linkend="catalog-pg-partitioned"><structname>pg_partitioned</structname></link></entry>

pg_partitioned seems like a slightly strange choice of name.  We have
no other catalog tables whose names end in "ed", so the use of a past
participle here is novel.  More generally, this patch seems like it's
suffering a bit of terminological confusion: while the catalog table
is called pg_partitioned, the relkind is RELKIND_PARTITION_REL.  And
get_relation_by_qualified_name() thinks that RELKIND_PARTITION_REL is
a "table", but getRelationDescription thinks it's a "partitioned
table".  I think we need to get all these bits and pieces on the same
page, or have some kind of consistent way of deciding what to do in
each case.  Maybe pg_partitioned_table, RELKIND_PARTITIONED_TABLE,
etc. for the internal stuff, but just "table" in user-facing messages.

Alternatively, I wonder if we should switch to calling this a
"partition root" rather than a "partitioned table".  It's not the
whole table, just the root.  And doesn't contain data - in fact it
probably shouldn't even have storage - so calling it a "table" might
be confusing.  But if we're using CREATE TABLE to create it in the
ifrst place, then calling it something other than a table later on is
also confusing.  Hmm.

+      <entry><structfield>partexprs</structfield></entry>

There's a certain symmetry between this and what we do for indexes,
but I'm wondering whether there's a use case for partitioning a table
by an expression rather than a column value.  I suppose if you've
already done the work, there's no harm in supporting it.

+[ PARTITION BY {RANGE | LIST} ( { <replaceable
class="parameter">column_name</replaceable> | ( <replaceable
class="parameter">expression</replaceable> ) } [ <replaceable
class="parameter">opclass</replaceable> ] [, ...] )

The spacing isn't right here.  Should say { RANGE | LIST } rather than
{RANGE|LIST}.  Similarly elsewhere.

+      thus created is called <firstterm>partitioned</firstterm> table.  Key
+      consists of an ordered list of column names and/or expressions when
+      using the <literal>RANGE</> method, whereas only a single column or
+      expression can be specified when using the <literal>LIST</> method.

Why do we support range partitioning with multiple columns, but list
partitioning only with a single column?

+      The type of a key column or an expresion must have an associated

Typo.

+     <para>
+      Currently, there are following limitations on definition of partitioned
+      tables: one cannot specify any UNIQUE, PRIMARY KEY, EXCLUDE and/or
+      FOREIGN KEY constraints.
+     </para>

But I presume you can define these constraints on the partitions;
that's probably worth mentioning.  I'd say something like this
"Partitioned tables do not support UNIQUE, PRIMARY, EXCLUDE, or
FOREIGN KEY constraints; however, you can define these constraints on
individual data partitions."

+    if (partexprs)
+        recordDependencyOnSingleRelExpr(&myself,
+                                        (Node *) partexprs,
+                                        RelationGetRelid(rel),
+                                        DEPENDENCY_NORMAL,
+                                        DEPENDENCY_IGNORE);

I don't think introducing a new DEPENDENCY_IGNORE type is a good idea
here.  Instead, you could just add an additional Boolean argument to
recordDependencyOnSingleRelExpr.  That seems less likely to create
bugs in unrelated portions of the code.

+    /*
+     * Override specified inheritance option, if relation is a partitioned
+     * table and it's a target of INSERT.
+     */
+    if (alsoSource)
+        rte->inh |= relid_is_partitioned(rte->relid);

This seems extremely unlikely to be the right way to do this.  We're
just doing parse analysis here, so depending on properties of the
table that are (or might be made to be) changeable is not good.  It's
also an extra catalog lookup whether the table is partitioned or not
(and whether rte->inh is already true or not). Furthermore, it seems
to go against the comment explaining what rte->inh is supposed to
mean, which says:
*        inh is TRUE for relation references that should be expanded to include*        inheritance children, if the
relhas any.  This *must* be FALSE for*        RTEs other than RTE_RELATION entries.
 

I am not sure what problem you're trying to solve here, but I suspect
this needs to be ripped out altogether or handled later, during
planning.  Similarly for the related change in addRangeTableEntry.

+    {PartitionedRelationId,        /* PARTEDRELID */
+        PartitionedRelidIndexId,
+        1,
+        {
+            Anum_pg_partitioned_partedrelid,
+            0,
+            0,
+            0
+        },
+        128

I'd probably cut the initial size of this down a bit.  Odds are good
that not all of the tables you access will be partitioned.  Of course
that assumes we'll avoid probing this syscache for non-partitioned
tables, but I think that's pretty important.

+            ereport(ERROR,
+                    (errcode(ERRCODE_INVALID_TABLE_DEFINITION),
+                     errmsg("cannot alter column named in partition key")));
+        else
+            ereport(ERROR,
+                    (errcode(ERRCODE_INVALID_TABLE_DEFINITION),
+                     errmsg("cannot alter column referenced in
partition key expression")));

Maybe "cannot alter type of column named in partition key"?  And
similarly for the other one.

+/*
+ * transformPartitionBy
+ *         Transform any expressions present in the partition key
+ */
+static PartitionBy *
+transformPartitionBy(Relation rel, PartitionBy *partitionby)

Shouldn't this be in src/backend/parser/parse_*.c instead of tablecmds.c?

Most of this (0001) looks pretty reasonable.  I'm sure that there is
more that needs fixing than what I've mentioned above, which is just
what I saw on an initial read-through, but overall I like the
approach.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Thanks a lot for taking a look at this.

On 2016/08/11 3:22, Robert Haas wrote:
> On Wed, Aug 10, 2016 at 7:09 AM, Amit Langote
> <Langote_Amit_f8@lab.ntt.co.jp> wrote:
>> Attached is the latest set of patches to implement declarative
>> partitioning.
> 
> Cool.  I would encourage you to give some thought to what is the least
> committable subset of these patches, and think about whether it can be
> reorganized to make that smaller.  Based on the descriptions, it
> sounds to me like the least committable subset is probably all of 0001
> - 0005 as one giant commit, and that's a lot of code.  Maybe there's
> no real way to come up with something more compact than that, but it's
> worth some thought.

I will consider this and try to come up with a minimal patch set covering
what is now 0001 - 0005.

>> 0001-Catalog-and-DDL-for-partition-key.patch
> 
> +      <entry><link
> linkend="catalog-pg-partitioned"><structname>pg_partitioned</structname></link></entry>
> 
> pg_partitioned seems like a slightly strange choice of name.  We have
> no other catalog tables whose names end in "ed", so the use of a past
> participle here is novel.  More generally, this patch seems like it's
> suffering a bit of terminological confusion: while the catalog table
> is called pg_partitioned, the relkind is RELKIND_PARTITION_REL.  And
> get_relation_by_qualified_name() thinks that RELKIND_PARTITION_REL is
> a "table", but getRelationDescription thinks it's a "partitioned
> table".  I think we need to get all these bits and pieces on the same
> page, or have some kind of consistent way of deciding what to do in
> each case.  Maybe pg_partitioned_table, RELKIND_PARTITIONED_TABLE,
> etc. for the internal stuff, but just "table" in user-facing messages.

Name pg_partitioned does sound a little strange now that you mention it.

I agree to make the terminology more consistent and to that end, your
suggestion to keep the user-facing term "table" and using
pg_partitioned_table and RELKIND_PARTITIONED_TABLE for internals sounds good.

> Alternatively, I wonder if we should switch to calling this a
> "partition root" rather than a "partitioned table".  It's not the
> whole table, just the root.  And doesn't contain data - in fact it
> probably shouldn't even have storage - so calling it a "table" might
> be confusing.  But if we're using CREATE TABLE to create it in the
> ifrst place, then calling it something other than a table later on is
> also confusing.  Hmm.

I think it makes sense to keep calling it a table because it has all the
logical properties of a table even though it will differ from a regular
table on the basis of physical implementation details such as that it does
not own physical storage.  Am I missing something?

> 
> +      <entry><structfield>partexprs</structfield></entry>
> 
> There's a certain symmetry between this and what we do for indexes,
> but I'm wondering whether there's a use case for partitioning a table
> by an expression rather than a column value.  I suppose if you've
> already done the work, there's no harm in supporting it.

Yeah, it's not a whole lot of code to manage expressions alongside simple
column references.

> +[ PARTITION BY {RANGE | LIST} ( { <replaceable
> class="parameter">column_name</replaceable> | ( <replaceable
> class="parameter">expression</replaceable> ) } [ <replaceable
> class="parameter">opclass</replaceable> ] [, ...] )
> 
> The spacing isn't right here.  Should say { RANGE | LIST } rather than
> {RANGE|LIST}.  Similarly elsewhere.

Will fix.

> +      thus created is called <firstterm>partitioned</firstterm> table.  Key
> +      consists of an ordered list of column names and/or expressions when
> +      using the <literal>RANGE</> method, whereas only a single column or
> +      expression can be specified when using the <literal>LIST</> method.
> 
> Why do we support range partitioning with multiple columns, but list
> partitioning only with a single column?

This is mostly because I didn't find any other database supporting it and
hence thought maybe there is not much use for it.

On the implementation side, I think it makes sense to think of ordering of
tuples (for a multi-column range key), where we compare a new tuple's
partition key columns one-by-one until we find the column such that its
value != the value of corresponding column in the range upper bound of a
partition (usually but not necessarily the last column of the key).  Once
we have determined which side of the bound the value was, we then perform
the same process with either the lower bound of the same partition or
upper bound of some partition in the other half as determined by binary
search logic.

Conversely, when determining the constraint on rows a range partition
contains, we emit a (keycol = lowerval) expression for all columns i =
0..j-1 where range.lowerval[i] = range.upperval[i] and then emit (keycol >
/ >= range.lowerval[j] and keycol < / <= upperval[j]) for the first column
j where lowerval[j] < upperval[j] and stop at that column (any further
columns are irrelevant).

When considering the same for list partitioning where we consider set-
membership of values (based on equality of a new tuple's value for the
column and a partition's list of values), implementing multi-column logic
does not seem as straightforward.  Also, it might make the optimizations
we recently discussed slightly more complicated to implement.  Although I
may be missing something.

> +      The type of a key column or an expresion must have an associated
> 
> Typo.

Will fix.

> 
> +     <para>
> +      Currently, there are following limitations on definition of partitioned
> +      tables: one cannot specify any UNIQUE, PRIMARY KEY, EXCLUDE and/or
> +      FOREIGN KEY constraints.
> +     </para>
> 
> But I presume you can define these constraints on the partitions;
> that's probably worth mentioning.  I'd say something like this
> "Partitioned tables do not support UNIQUE, PRIMARY, EXCLUDE, or
> FOREIGN KEY constraints; however, you can define these constraints on
> individual data partitions."

Okay, will mention that.

> +    if (partexprs)
> +        recordDependencyOnSingleRelExpr(&myself,
> +                                        (Node *) partexprs,
> +                                        RelationGetRelid(rel),
> +                                        DEPENDENCY_NORMAL,
> +                                        DEPENDENCY_IGNORE);
> 
> I don't think introducing a new DEPENDENCY_IGNORE type is a good idea
> here.  Instead, you could just add an additional Boolean argument to
> recordDependencyOnSingleRelExpr.  That seems less likely to create
> bugs in unrelated portions of the code.

I did consider a Boolean argument instead of a new DependencyType value,
however it felt a bit strange to pass a valid value for the fifth argument
(self_behavior) and then ask using a separate parameter that it (a
self-dependency) is to be ignored.  By the way, no pg_depend entry is
created on such a call, so the effect of the new type's usage seems
localized to me. Thoughts?

> 
> +    /*
> +     * Override specified inheritance option, if relation is a partitioned
> +     * table and it's a target of INSERT.
> +     */
> +    if (alsoSource)
> +        rte->inh |= relid_is_partitioned(rte->relid);
> 
> This seems extremely unlikely to be the right way to do this.  We're
> just doing parse analysis here, so depending on properties of the
> table that are (or might be made to be) changeable is not good.  It's
> also an extra catalog lookup whether the table is partitioned or not
> (and whether rte->inh is already true or not). Furthermore, it seems
> to go against the comment explaining what rte->inh is supposed to
> mean, which says:
> 
>  *        inh is TRUE for relation references that should be expanded to include
>  *        inheritance children, if the rel has any.  This *must* be FALSE for
>  *        RTEs other than RTE_RELATION entries.
> 
> I am not sure what problem you're trying to solve here, but I suspect
> this needs to be ripped out altogether or handled later, during
> planning.  Similarly for the related change in addRangeTableEntry.

Okay, I will see how this could rather be done within the planner.  My
intention here is to keep the code churn low by not modifying a lot of
places in the code where rte->inh is checked to turn inheritance on or
off.  I agree though that it's a bit ugly and perhaps wrong.

> +    {PartitionedRelationId,        /* PARTEDRELID */
> +        PartitionedRelidIndexId,
> +        1,
> +        {
> +            Anum_pg_partitioned_partedrelid,
> +            0,
> +            0,
> +            0
> +        },
> +        128
> 
> I'd probably cut the initial size of this down a bit.  Odds are good
> that not all of the tables you access will be partitioned.  Of course
> that assumes we'll avoid probing this syscache for non-partitioned
> tables, but I think that's pretty important.

Okay, I will turn that down to 64 maybe.

> 
> +            ereport(ERROR,
> +                    (errcode(ERRCODE_INVALID_TABLE_DEFINITION),
> +                     errmsg("cannot alter column named in partition key")));
> +        else
> +            ereport(ERROR,
> +                    (errcode(ERRCODE_INVALID_TABLE_DEFINITION),
> +                     errmsg("cannot alter column referenced in
> partition key expression")));
> 
> Maybe "cannot alter type of column named in partition key"?  And
> similarly for the other one.

Will fix.

> 
> +/*
> + * transformPartitionBy
> + *         Transform any expressions present in the partition key
> + */
> +static PartitionBy *
> +transformPartitionBy(Relation rel, PartitionBy *partitionby)
> 
> Shouldn't this be in src/backend/parser/parse_*.c instead of tablecmds.c?

It can be performed only after the new relation is opened which can be
only after we have called heap_create_with_catalog() which is in
tablecmds.c: DefineRelation().  I think that's because we call
transformExpr() on partition expressions which expects to be able to see
the table's columns.  Other callers such as transformIndexStmt and
transformAlterTableStmt can do transformations within parse_utilcmds.c
because they can open the relation there.

> 
> Most of this (0001) looks pretty reasonable.  I'm sure that there is
> more that needs fixing than what I've mentioned above, which is just
> what I saw on an initial read-through, but overall I like the
> approach.
> 

Okay, thanks.  I will post the updated patches soon.

Thanks,
Amit

On Wed, Aug 10, 2016 at 7:09 AM, Amit Langote
<Langote_Amit_f8@lab.ntt.co.jp> wrote:
> 0002-psql-and-pg_dump-support-for-partitioned-tables.patch

+    if (pset.sversion >= 90600 && tableinfo.relkind == 'P')

Version check is redundant, right?

+) PARTITION BY RANGE ((a+b));
+\d describe_range_key
+Partitioned table "public.describe_range_key"
+ Column |  Type   | Modifiers
+--------+---------+-----------
+ a      | integer |
+ b      | integer |
+Partition Key: PARTITION BY RANGE (((a + b)))

I understand that it's probably difficult not to end up with two sets
of parentheses here, but can we avoid ending up with three sets?

Also, I wonder if pg_get_partkeydef() should omit "PARTITION BY" and
pg_dump can add that part back.  Then this could say:

Partition Key: RANGE ((a + b))

...which seems a good deal more natural than what you have now.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Wed, Aug 10, 2016 at 7:09 AM, Amit Langote
<Langote_Amit_f8@lab.ntt.co.jp> wrote:
> 0003-Catalog-and-DDL-for-partition-bounds.patch
>
> Partition DDL includes both a way to create new partition and "attach" an
> existing table as a partition of parent partitioned table.  Attempt to
> drop a partition using DROP TABLE causes an error. Instead a partition
> needs first to be "detached" from parent partitioned table.  On the other
> hand, dropping the parent drops all the partitions if CASCADE is specified.

Hmm, I don't think I like this.  Why should it be necessary to detach
a partition before dropping it?  That seems like an unnecessary step.
    [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY
IMMEDIATE ]
+</synopsis>

Unnecessary hunk.

+     <para>
+      If this table is a partition, one cannot perform <literal>DROP
NOT NULL</>
+      on a column if it is marked not null in the parent table.
+      not null.
+     </para>

Sentence fragment.

+     <para>
+      Note that unlike the <literal>ATTACH PARTITION</> command, a partition
+      being detached can be itself partitioned.  In that case, it continues
+      to exist as such.
+     </para>

This is another restriction I don't understand.  Why can't I attach a
partitioned table?

+        indicate that descendant tables are included.  Note that whether
+        <literal>ONLY</> or <literal>*</> is specified has no effect in case
+        of a partitioned table; descendant tables (in this case, partitions)
+        are always included.

Ugh, why?  I think this should work exactly the same way for
partitioned tables that it does for any other inheritance hierarchy.
Sure, you'll get no rows, but who cares?

+CREATE FOREIGN TABLE measurement_y2016m07
+    PARTITION OF measurement FOR VALUES START ('2016-07-01') END
('2016-08-01');
+    SERVER server_07;

Extra semicolon?

+      A partition cannot have columns other than those inherited from the
+      parent.  That includes the <structfield>oid</> column, which can be

I think experience suggests that this is a good restriction, but then
why does the syntax synopsis indicate that PARTITION BY can be
specified along with column definitions?  Similarly for CREATE FOREIGN
TABLE.

+      When specifying for a table being created as partition, one needs to
+      use column names from the parent table as part of the key.

This is not very clear.

-       /* Remove NO INHERIT flag if rel is a partitioned table */
-       if (relid_is_partitioned(relid))
+       /* Discard NO INHERIT, if relation is a partitioned table or a
partition */
+       if (relid_is_partitioned(relid) || relid_is_partition(relid))               is_no_inherit = false;

It might be right to disallow NO INHERIT in this case, but I don't
think it can be right to just silently ignore it.

+ * Not flushed from the cache by RelationClearRelation() unless changed because
+ * of addition or removal of partitions.

This seems unlikely to be safe, unless I'm missing something.

+       form = (Form_pg_inherits) GETSTRUCT(tuple);
+
+       systable_endscan(scan);
+       heap_close(catalogRelation, AccessShareLock);
+
+       return form->inhparent;

This is unsafe.  After systable_endscan, it is no longer OK to access
form->inhparent.

Try building with CLOBBER_CACHE_ALWAYS to find other cache flush hazards.

There should probably be a note in the function header comment that it
is unsafe to use this for an inheritance child that is not a
partition, because there could be more than one parent in that case.
Or maybe the whole idea of this function just isn't very sound...

+static List *
+get_partitions(Oid relid, int lockmode)
+{
+       return find_inheritance_children(relid, lockmode);
+}

What's the point?  If we're going to have a wrapper here at all, then
shouldn't it have a name that matches the existing convention - e.g.
find_partitions() or find_child_partitions()?  But I think you might
as well just use find_inheritance_children() directly.

+                * Happens when we have created the pg_inherits entry
but not the
+                * pg_partition entry yet.

Why do we ever allow the flow of control to reach this point while we
are in such an intermediate state?

+free_partition_info(PartitionInfoData **p, int num)

Seems very error-prone.  Isn't this why MemoryContextReset was invented?

+relid_is_partition(Oid relid)
+{
+       return SearchSysCacheExists1(PARTRELID, ObjectIdGetDatum(relid));
+}

This is used in a lot of places, and the overhead of checking it in
all of those places is not necessarily nil.  Syscache lookups aren't
free.  What if we didn't create a new catalog for this and instead
just added a relpartitionbound attribute to pg_class?  It seems a bit
silly to have a whole extra catalog to store one extra column...
       /*
+        * If this foreign table is a partition, check that the FDW supports
+        * insert.
+        */
+       if (stmt->base.partbound != NULL)
+       {
+               FdwRoutine *fdw_routine;
+
+               fdw_routine = GetFdwRoutine(fdw->fdwhandler);
+               if (fdw_routine->ExecForeignInsert == NULL)
+                       ereport(ERROR,
+                                       (errcode(ERRCODE_FDW_NO_SCHEMAS),
+                                        errmsg("cannot create foreign
table as partition"),
+                                        errdetail("foreign-data
wrapper \"%s\" does not support insert",
+                                                       fdw->fdwname)));
+       }

Why?  This seems like an entirely arbitrary prohibition.  If inserts
aren't supported, then they'll fail at runtime.  Same for updates or
deletes, for which you have not added checks.  I think you should just
remove this.

+               /* Force inheritance recursion, if partitioned table. */
+               if (recurse || relid_is_partitioned(myrelid))

Disagree with this, too.  There's no reason for partitioned tables to
be special in this way.  Similarly, disagree with all of the places
that do something similar.

-                               errmsg("column \"%s\" in child table
must be marked NOT NULL",
-                                          attributeName)));
+                               errmsg("column \"%s\" in %s table must
be marked NOT NULL",
+                                          attributeName,
+                                          is_attach_partition ?
"source" : "child")));

You have a few of these; they cause problems for translators, because
different languages have different word ordering.  Repeat the entire
message instead: is_attach_partition ? "column \"%s\" in source table
must be marked NOT NULL" : "column \"%s\" in child table must be
marked NOT NULL".

+-- XXX add ownership tests

So do that.  :-)

+ERROR:  column "b" is not null in parent
+HINT:  Please drop not null in the parent instead

Hmm.   That hint doesn't seem like project style, and I'm not sure
that it really makes sense to issue such a hint anyway.  Who knows
whether that is the right thing to do?  I think you should somehow be
complaining about the fact that this is a partition, rather than
complaining about the fact that the column is NOT NULL in the parent.
Are we insisting that the flags match exactly, or only that the child
may not allow nulls unless the parent does?

+ERROR:  new partition's list of values overlaps with partition
"lpart1" of "list_parted"

Maybe just:

ERROR: partitions must not overlap
-or-
ERROR: partition "%s" would overlap partition "%s"

As before, this is just an initial read-through, so apologies for
whatever I may have missed.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 2016/08/17 14:33, Ashutosh Bapat wrote:
>> +relid_is_partition(Oid relid)
>> +{
>> +       return SearchSysCacheExists1(PARTRELID, ObjectIdGetDatum(relid));
>> +}
>>
>> This is used in a lot of places, and the overhead of checking it in
>> all of those places is not necessarily nil.  Syscache lookups aren't
>> free.  What if we didn't create a new catalog for this and instead
>> just added a relpartitionbound attribute to pg_class?  It seems a bit
>> silly to have a whole extra catalog to store one extra column...
>>
> It looks like in most of the places where this function is called it's
> using relid_is_partition(RelationGetRelid(rel)). Instead probably we should
> check existence of rd_partdesc or rd_partkey within Relation() and make
> sure that those members are always set for a partitioned table. That will
> avoid cache lookup and may give better performance.

It seems you are talking about a *partitioned* relation here, whereas
relid_is_partition() is to trying to check if a relation is *partition* by
looking up the pg_partition catalog (or the associated cache).  For the
former, the test you suggest or rd_rel->relkind ==
RELKIND_PARTITIONED_TABLE test is enough.

I am slightly tempted to eliminate the pg_partition catalog and associated
syscache altogether and add a column to pg_class as Robert suggested.
That way, all relid_is_partition() calls will be replaced by
rel->rd_partbound != NULL check.  But one potential problem with that
approach is that now whenever a parent relation is opened, all the
partition relations must be opened to get the partbound value (to form the
PartitionDesc to be stored in parent relation's rd_partdesc).  Whereas
currently, we just look up the pg_partition catalog (or the associated
cache) for every partition and that gets us the partbound.

> That brings up another question. Can we have rd_partdesc non null and
> rd_partkey null or vice-versa. If not, should we club those into a single
> structure like Partition (similar to Relation)?

It's true that rd_partkey and rd_partdesc are both either NULL or
non-NULL, so combining them into a single struct is an idea worth considering.

Thanks,
Amit

On Wed, Aug 17, 2016 at 2:21 AM, Amit Langote
<Langote_Amit_f8@lab.ntt.co.jp> wrote:
> I am slightly tempted to eliminate the pg_partition catalog and associated
> syscache altogether and add a column to pg_class as Robert suggested.
> That way, all relid_is_partition() calls will be replaced by
> rel->rd_partbound != NULL check.  But one potential problem with that
> approach is that now whenever a parent relation is opened, all the
> partition relations must be opened to get the partbound value (to form the
> PartitionDesc to be stored in parent relation's rd_partdesc).  Whereas
> currently, we just look up the pg_partition catalog (or the associated
> cache) for every partition and that gets us the partbound.

Well, you could just look up the pg_class row without opening the
relation, too.  There is a system cache on pg_class.oid, after all.  I
think the issue is whether it's safe to read either one of those
things without a lock on the child relation.  If altering the
partitioning information for a relation requires holding only
AccessExclusiveLock on that relation, and no lock on the parent, then
you really can't read the information for any child relation without
taking at least AccessShareLock.  Otherwise, it might change under
you, and that would be bad.

I'm inclined to think that changing the partitioning information for a
child is going to require AccessExclusiveLock on both the child and
the parent.  That seems unfortunate from a concurrency point of view,
but we may be stuck with it: suppose you require only
ShareUpdateExclusiveLock on the parent.  Well, then a concurrent read
transaction might see the partition boundaries change when it does a
relcache rebuild, which would cause it to suddenly start expecting the
data to be in a different plan in mid-transaction, perhaps even in
mid-scan.  Maybe that's survivable with really careful coding, but it
seems like it's probably a bad thing.  For example, it would mean that
the executor would be unable to rely on the partitioning information
in the relcache remaining stable underneath it.  Moreover, the
relcache is always going to be scanned with the most recent possible
MVCC snapshot, but the transaction snapshot may be older, so such a
system creates all sorts of nasty possibilities for there to be skew
between the snapshot being used to via the data and the snapshot being
used to read the metadata that says where the data is.

This may need some more thought, but if we go with that approach of
requiring an AccessExclusiveLock on both parent and child, then it
seems to me that maybe we should consider the partitioning information
to be a property of the parent rather than the child.  Just take all
the partitioning information for all children and put it in one big
node tree and store it in the pg_class or pg_partition_root entry for
the parent as one big ol' varlena.  Now you can open the parent and
get all of the partitioning information for all of the children
without needing any lock on any child, and that's *really* good,
because it means that some day we might be able to do partition
elimination before locking any of the children!  That would be
excellent.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 2016/08/22 13:51, Ashutosh Bapat wrote:
> The parent-child relationship of multi-level partitioned tables is not
> retained when creating the AppendRelInfo nodes. We create RelOptInfo nodes
> for all the leaf and intermediate tables. The AppendRelInfo nodes created
> for these RelOptInfos set the topmost table as the parent of all the leaf
> and child tables. Since partitioning scheme/s at each level is/are
> associated with the parent/s at that level, we loose information about the
> immediate parents and thus it becomes difficult to identify which leaf node
> falls where in the partition hierarchy. This stops us from doing any
> lump-sum partition pruning where we can eliminate all the partitions under
> a given parent-partition if that parent-partition gets pruned. It also
> restricts partition-wise join technique from being applied to partial
> partition hierarchy when the whole partitioning scheme of joining tables
> does not match. Maintaining a RelOptInfo hierarchy should not create
> corresponding Append (all kinds) plan hierarchy since
> accumulate_append_subpath() flattens any such hierarchy while creating
> paths. Can you please consider this point in your upcoming patch?

I agree.  So there seem to be two things here:  a) when expanding a
partitioned table inheritance set, do it recursively such that resulting
AppendRelInfos preserve *immediate* parent-child relationship info.  b)
when accumulating append subpaths, do not flatten a subpath that is itself
an append when ((AppendPath *) subpath)->path.parent is a RelOptInfo with
non-NULL partitioning info.  Is the latter somehow necessary for
pairwise-join considerations?

I think I can manage to squeeze in (a) in the next version patch and will
also start working on (b), mainly the part about RelOptInfo getting some
partitioning info.

Thanks,
Amit

On 2016/08/18 5:23, Robert Haas wrote:
> On Wed, Aug 17, 2016 at 2:21 AM, Amit Langote
> <Langote_Amit_f8@lab.ntt.co.jp> wrote:
>> I am slightly tempted to eliminate the pg_partition catalog and associated
>> syscache altogether and add a column to pg_class as Robert suggested.
>> That way, all relid_is_partition() calls will be replaced by
>> rel->rd_partbound != NULL check.  But one potential problem with that
>> approach is that now whenever a parent relation is opened, all the
>> partition relations must be opened to get the partbound value (to form the
>> PartitionDesc to be stored in parent relation's rd_partdesc).  Whereas
>> currently, we just look up the pg_partition catalog (or the associated
>> cache) for every partition and that gets us the partbound.
> 
> Well, you could just look up the pg_class row without opening the
> relation, too.  There is a system cache on pg_class.oid, after all.  I

Yes, I somehow didn't think of that.

> think the issue is whether it's safe to read either one of those
> things without a lock on the child relation.  If altering the
> partitioning information for a relation requires holding only
> AccessExclusiveLock on that relation, and no lock on the parent, then
> you really can't read the information for any child relation without
> taking at least AccessShareLock.  Otherwise, it might change under
> you, and that would be bad.

I'd imagine this won't be a problem because we take an AccessExclusiveLock
on the parent when adding/removing a partition.

> I'm inclined to think that changing the partitioning information for a
> child is going to require AccessExclusiveLock on both the child and
> the parent.  That seems unfortunate from a concurrency point of view,
> but we may be stuck with it: suppose you require only
> ShareUpdateExclusiveLock on the parent.  Well, then a concurrent read
> transaction might see the partition boundaries change when it does a
> relcache rebuild, which would cause it to suddenly start expecting the
> data to be in a different plan in mid-transaction, perhaps even in
> mid-scan.  Maybe that's survivable with really careful coding, but it
> seems like it's probably a bad thing.  For example, it would mean that
> the executor would be unable to rely on the partitioning information
> in the relcache remaining stable underneath it.  Moreover, the
> relcache is always going to be scanned with the most recent possible
> MVCC snapshot, but the transaction snapshot may be older, so such a
> system creates all sorts of nasty possibilities for there to be skew
> between the snapshot being used to via the data and the snapshot being
> used to read the metadata that says where the data is.

We do take a lock on the parent because we would be changing its partition
descriptor (relcache).  I changed MergeAttributes() such that an
AccessExclusiveLock instead of ShareUpdateExclusiveLock is taken if the
parent is a partitioned table.

> This may need some more thought, but if we go with that approach of
> requiring an AccessExclusiveLock on both parent and child, then it
> seems to me that maybe we should consider the partitioning information
> to be a property of the parent rather than the child.  Just take all
> the partitioning information for all children and put it in one big
> node tree and store it in the pg_class or pg_partition_root entry for
> the parent as one big ol' varlena.  Now you can open the parent and
> get all of the partitioning information for all of the children
> without needing any lock on any child, and that's *really* good,
> because it means that some day we might be able to do partition
> elimination before locking any of the children!  That would be
> excellent.

If we need an AccessExclusiveLock on parent to add/remove a partition
(IOW, changing that child table's partitioning information), then do we
need to lock the individual partitions when reading partition's
information?  I mean to ask why the simple syscache look-ups to get each
partition's bound wouldn't do.

Thanks,
Amit

Sorry it took me a while to reply.  Attached updated patches including the
review comments on 0001 at [1].

On 2016/08/17 3:54, Robert Haas wrote:
> On Wed, Aug 10, 2016 at 7:09 AM, Amit Langote
> <Langote_Amit_f8@lab.ntt.co.jp> wrote:
>> 0002-psql-and-pg_dump-support-for-partitioned-tables.patch
>
> +    if (pset.sversion >= 90600 && tableinfo.relkind == 'P')
>
> Version check is redundant, right?

Yep, fixed.

> +) PARTITION BY RANGE ((a+b));
> +\d describe_range_key
> +Partitioned table "public.describe_range_key"
> + Column |  Type   | Modifiers
> +--------+---------+-----------
> + a      | integer |
> + b      | integer |
> +Partition Key: PARTITION BY RANGE (((a + b)))
>
> I understand that it's probably difficult not to end up with two sets
> of parentheses here, but can we avoid ending up with three sets?

Fixed.  This code was copy-pasted from the index code which has other
considerations for adding surrounding parentheses which don't apply to the
partition key code.

> Also, I wonder if pg_get_partkeydef() should omit "PARTITION BY" and
> pg_dump can add that part back.  Then this could say:
>
> Partition Key: RANGE ((a + b))
>
> ...which seems a good deal more natural than what you have now.

Agreed, so done that way.

>> 0003-Catalog-and-DDL-for-partition-bounds.patch
>>
>> Partition DDL includes both a way to create new partition and "attach" an
>> existing table as a partition of parent partitioned table.  Attempt to
>> drop a partition using DROP TABLE causes an error. Instead a partition
>> needs first to be "detached" from parent partitioned table.  On the other
>> hand, dropping the parent drops all the partitions if CASCADE is specified.
>
> Hmm, I don't think I like this.  Why should it be necessary to detach
> a partition before dropping it?  That seems like an unnecessary step.

I thought we had better lock the parent table when removing one of its
partitions and it seemed a bit odd to lock the parent table when dropping
a partition using DROP TABLE?  OTOH, with ALTER TABLE parent DETACH
PARTITION, the parent table is locked anyway.

>      [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY
> IMMEDIATE ]
> +
>  </synopsis>
>
> Unnecessary hunk.
>
> +     <para>
> +      If this table is a partition, one cannot perform <literal>DROP
> NOT NULL</>
> +      on a column if it is marked not null in the parent table.
> +      not null.
> +     </para>
>
> Sentence fragment.

Fixed.

>
> +     <para>
> +      Note that unlike the <literal>ATTACH PARTITION</> command, a partition
> +      being detached can be itself partitioned.  In that case, it continues
> +      to exist as such.
> +     </para>
>
> This is another restriction I don't understand.  Why can't I attach a
> partitioned table?

I removed this restriction.

ATExecAttachPartition() adds a AT work queue entry for the table being
attached to perform a heap scan in the rewrite phase.  The scan is done
for checking that no row violates the partition boundary condition.  When
attaching a partitioned table as partition, multiple AT work queue entries
are now added - one for each leaf partition of the table being attached.

> +        indicate that descendant tables are included.  Note that whether
> +        <literal>ONLY</> or <literal>*</> is specified has no effect in case
> +        of a partitioned table; descendant tables (in this case, partitions)
> +        are always included.
>
> Ugh, why?  I think this should work exactly the same way for
> partitioned tables that it does for any other inheritance hierarchy.
> Sure, you'll get no rows, but who cares?

Agreed, done that way.

> +CREATE FOREIGN TABLE measurement_y2016m07
> +    PARTITION OF measurement FOR VALUES START ('2016-07-01') END
> ('2016-08-01');
> +    SERVER server_07;
>
> Extra semicolon?

Fixed.

>
> +      A partition cannot have columns other than those inherited from the
> +      parent.  That includes the <structfield>oid</> column, which can be
>
> I think experience suggests that this is a good restriction, but then
> why does the syntax synopsis indicate that PARTITION BY can be
> specified along with column definitions?  Similarly for CREATE FOREIGN
> TABLE.

The syntax synopsis of CREATE TABLE ... PARTITION OF indicates that a list
of column WITH OPTION and/or table_constraint can be specified.  It does
not allow column definitions.

In this case, inherited columns will be listed in the PARTITION BY clause.

Do you mean that the CREATE TABLE ... PARTITION OF syntax should allow
column definitions just like INHERITS does and unlike regular inheritance,
throw error if columns other than those to be merged are found?

>
> +      When specifying for a table being created as partition, one needs to
> +      use column names from the parent table as part of the key.
>
> This is not very clear.

Sentence removed because it may be clear from the context that inherited
columns are to be used for the partition key.

> -       /* Remove NO INHERIT flag if rel is a partitioned table */
> -       if (relid_is_partitioned(relid))
> +       /* Discard NO INHERIT, if relation is a partitioned table or a
> partition */
> +       if (relid_is_partitioned(relid) || relid_is_partition(relid))
>                 is_no_inherit = false;
>
> It might be right to disallow NO INHERIT in this case, but I don't
> think it can be right to just silently ignore it.

OK, I changed this to instead throw an error if a NO INHERIT check
constraint is added to a partitioned table or a partition.

> + * Not flushed from the cache by RelationClearRelation() unless changed because
> + * of addition or removal of partitions.
>
> This seems unlikely to be safe, unless I'm missing something.

Like TupleDesc, a table's PartitionDesc is preserved across relcache
rebuilds.  PartitionDesc consists of arrays of OIDs and partition bounds
for a table's immediate partitions.  It can only change by adding/removing
partitions to/from the table which requires an exclusive lock on it.
Since this data can grow arbitrarily big it seemed better to not have to
copy it around, so a direct pointer to the relcache field (rd_partdesc) is
given to callers.  relcache rebuilds that do not logically change
PartitionDesc leave it intact so that some user of it is not left with a
dangling pointer.  Am I missing something?

>
> +       form = (Form_pg_inherits) GETSTRUCT(tuple);
> +
> +       systable_endscan(scan);
> +       heap_close(catalogRelation, AccessShareLock);
> +
> +       return form->inhparent;
>
> This is unsafe.  After systable_endscan, it is no longer OK to access
> form->inhparent.
>
> Try building with CLOBBER_CACHE_ALWAYS to find other cache flush hazards.
>
> There should probably be a note in the function header comment that it
> is unsafe to use this for an inheritance child that is not a
> partition, because there could be more than one parent in that case.
> Or maybe the whole idea of this function just isn't very sound...

Fixed unsafe coding and added a comment to the function saying it should
be called on tables known to be partitions.

>
> +static List *
> +get_partitions(Oid relid, int lockmode)
> +{
> +       return find_inheritance_children(relid, lockmode);
> +}
>
> What's the point?  If we're going to have a wrapper here at all, then
> shouldn't it have a name that matches the existing convention - e.g.
> find_partitions() or find_child_partitions()?  But I think you might
> as well just use find_inheritance_children() directly.

OK, changed to just use find_inheritance_children() directly.

>
> +                * Happens when we have created the pg_inherits entry
> but not the
> +                * pg_partition entry yet.
>
> Why do we ever allow the flow of control to reach this point while we
> are in such an intermediate state?

Fixed to prevent this from happening.

>
> +free_partition_info(PartitionInfoData **p, int num)
>
> Seems very error-prone.  Isn't this why MemoryContextReset was invented?

Got rid of this function.

>
> +relid_is_partition(Oid relid)
> +{
> +       return SearchSysCacheExists1(PARTRELID, ObjectIdGetDatum(relid));
> +}
>
> This is used in a lot of places, and the overhead of checking it in
> all of those places is not necessarily nil.  Syscache lookups aren't
> free.  What if we didn't create a new catalog for this and instead
> just added a relpartitionbound attribute to pg_class?  It seems a bit
> silly to have a whole extra catalog to store one extra column...

OK, I got rid of the pg_partition catalog and added a pg_class attribute
for storing partition bound.  Because the newly added attribute is a
pg_node_tree and hence not readily accessible without a heap_getattr()
call, I added a boolean relispartition as well. Now all the
relid_is_partition() calls have been replaced by checks using
relation->rd_rel->relispartition.

>
>         /*
> +        * If this foreign table is a partition, check that the FDW supports
> +        * insert.
> +        */
> +       if (stmt->base.partbound != NULL)
> +       {
> +               FdwRoutine *fdw_routine;
> +
> +               fdw_routine = GetFdwRoutine(fdw->fdwhandler);
> +               if (fdw_routine->ExecForeignInsert == NULL)
> +                       ereport(ERROR,
> +                                       (errcode(ERRCODE_FDW_NO_SCHEMAS),
> +                                        errmsg("cannot create foreign
> table as partition"),
> +                                        errdetail("foreign-data
> wrapper \"%s\" does not support insert",
> +                                                       fdw->fdwname)));
> +       }
>
> Why?  This seems like an entirely arbitrary prohibition.  If inserts
> aren't supported, then they'll fail at runtime.  Same for updates or
> deletes, for which you have not added checks.  I think you should just
> remove this.

OK, removed this check.

Instead, ExecInitModifyTable()/BeginCopy() call CheckValidResultRel() for
every leaf partition to check that they are ready for CMD_INSERT.

>
> +               /* Force inheritance recursion, if partitioned table. */
> +               if (recurse || relid_is_partitioned(myrelid))
>
> Disagree with this, too.  There's no reason for partitioned tables to
> be special in this way.  Similarly, disagree with all of the places
> that do something similar.

Removed the forced recursion bit here and a few other places.

>
> -                               errmsg("column \"%s\" in child table
> must be marked NOT NULL",
> -                                          attributeName)));
> +                               errmsg("column \"%s\" in %s table must
> be marked NOT NULL",
> +                                          attributeName,
> +                                          is_attach_partition ?
> "source" : "child")));
>
> You have a few of these; they cause problems for translators, because
> different languages have different word ordering.  Repeat the entire
> message instead: is_attach_partition ? "column \"%s\" in source table
> must be marked NOT NULL" : "column \"%s\" in child table must be
> marked NOT NULL".

Changed so that the entire message is repeated.

>
> +-- XXX add ownership tests
>
> So do that.  :-)

Done, sorry about that.

>
> +ERROR:  column "b" is not null in parent
> +HINT:  Please drop not null in the parent instead
>
> Hmm.   That hint doesn't seem like project style, and I'm not sure
> that it really makes sense to issue such a hint anyway.  Who knows
> whether that is the right thing to do?  I think you should somehow be
> complaining about the fact that this is a partition, rather than
> complaining about the fact that the column is NOT NULL in the parent.
> Are we insisting that the flags match exactly, or only that the child
> may not allow nulls unless the parent does?

I think the latter.  It is assumed that all the parent's constraints are
present in a child table, because in ExecInsert()/CopyFrom() we perform
ExecConstraints() using the child relation even if the actual insert was
on the parent.  Also, if an inherited NOT NULL constraint on a child's
column is dropped irrespective of parent's, selecting a parent's NOT NULL
column might return nulls from the child table that no longer has the
constraint.

I recently came across a related proposal whereby dropping *inherited* NOT
NULL  from child tables will be prevented.  Problems in letting it be be
dropped are mentioned here:

https://www.postgresql.org/message-id/21633.1448383428@sss.pgh.pa.us

That proposal is probably being reworked such that NOT NULL constraints
get a pg_constraint entry with proper accounting of inheritance count.

> +ERROR:  new partition's list of values overlaps with partition
> "lpart1" of "list_parted"
>
> Maybe just:
>
> ERROR: partitions must not overlap
> -or-
> ERROR: partition "%s" would overlap partition "%s"

OK, I changed to the second message.

>
> As before, this is just an initial read-through, so apologies for
> whatever I may have missed.

Thanks a lot for the review.

By the way, I am still working on the following items and will be included
in the next version of the patch.

* Fix internal representation of list partition bounds to be more efficient
* Add PartitionOptInfo

As mentioned in [2], I have refactored the inheritance expansion code
within optimizer so that a partitioned table's inheritance hierarchy is
preserved in resulting AppendRelInfos (patch 0005).  One immediate benefit
of that is that if constraint exclusion determines that an intermediate
partition is to be excluded then all partitions underneath it are excluded
automatically.  With the current flattened model, all partitions in that
subtree would have been processed and excluded one-by-one.

Regards,
Amit

[1]
https://www.postgresql.org/message-id/CA%2BTgmoZ008qTgd_Qg6_oZb3i0mOYrS6MdhncwgcqPKahixjarg%40mail.gmail.com
[2]
https://www.postgresql.org/message-id/f2a9592a-17e9-4c6a-e021-03b802195ce7%40lab.ntt.co.jp

On Fri, Aug 26, 2016 at 1:33 PM, Amit Langote
<Langote_Amit_f8@lab.ntt.co.jp> wrote:
> We do take a lock on the parent because we would be changing its partition
> descriptor (relcache).  I changed MergeAttributes() such that an
> AccessExclusiveLock instead of ShareUpdateExclusiveLock is taken if the
> parent is a partitioned table.

Hmm, that seems both good and bad.  On the good side, as mentioned,
being able to rely on the partition descriptor not changing under us
makes this sort of thing much easier to reason about.  On the bad
side, it isn't good for this feature to have worse concurrency than
regular inheritance.  Not sure what to do about this.

> If we need an AccessExclusiveLock on parent to add/remove a partition
> (IOW, changing that child table's partitioning information), then do we
> need to lock the individual partitions when reading partition's
> information?  I mean to ask why the simple syscache look-ups to get each
> partition's bound wouldn't do.

Well, if X can't be changed without having an AccessExclusiveLock on
the parent, then an AccessShareLock on the parent is sufficient to
read X, right?  Because those lock modes conflict.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 2016/08/29 20:53, Robert Haas wrote:
> On Fri, Aug 26, 2016 at 1:33 PM, Amit Langote
> <Langote_Amit_f8@lab.ntt.co.jp> wrote:
>> We do take a lock on the parent because we would be changing its partition
>> descriptor (relcache).  I changed MergeAttributes() such that an
>> AccessExclusiveLock instead of ShareUpdateExclusiveLock is taken if the
>> parent is a partitioned table.
> 
> Hmm, that seems both good and bad.  On the good side, as mentioned,
> being able to rely on the partition descriptor not changing under us
> makes this sort of thing much easier to reason about.  On the bad
> side, it isn't good for this feature to have worse concurrency than
> regular inheritance.  Not sure what to do about this.
>
>> If we need an AccessExclusiveLock on parent to add/remove a partition
>> (IOW, changing that child table's partitioning information), then do we
>> need to lock the individual partitions when reading partition's
>> information?  I mean to ask why the simple syscache look-ups to get each
>> partition's bound wouldn't do.
> 
> Well, if X can't be changed without having an AccessExclusiveLock on
> the parent, then an AccessShareLock on the parent is sufficient to
> read X, right?  Because those lock modes conflict.

Yes.  And hence we can proceed with performing partition elimination
before locking any of children.  Lock on parent (AccessShareLock) will
prevent any of existing partitions to be removed and any new partitions to
be added because those operations require AccessExclusiveLock on the
parent.  What I was trying to understand is why this would not be possible
with a design where partition bound is stored in the catalog as a property
of individual partitions instead of a design where we store collection of
partition bounds as a property of the parent.

Thanks,
Amit

On Wed, Aug 31, 2016 at 12:37 PM, Amit Langote
<Langote_Amit_f8@lab.ntt.co.jp> wrote:
>>> If we need an AccessExclusiveLock on parent to add/remove a partition
>>> (IOW, changing that child table's partitioning information), then do we
>>> need to lock the individual partitions when reading partition's
>>> information?  I mean to ask why the simple syscache look-ups to get each
>>> partition's bound wouldn't do.
>>
>> Well, if X can't be changed without having an AccessExclusiveLock on
>> the parent, then an AccessShareLock on the parent is sufficient to
>> read X, right?  Because those lock modes conflict.
>
> Yes.  And hence we can proceed with performing partition elimination
> before locking any of children.  Lock on parent (AccessShareLock) will
> prevent any of existing partitions to be removed and any new partitions to
> be added because those operations require AccessExclusiveLock on the
> parent.

Agreed.

> What I was trying to understand is why this would not be possible
> with a design where partition bound is stored in the catalog as a property
> of individual partitions instead of a design where we store collection of
> partition bounds as a property of the parent.

From the point of view of feasibility, I don't think it matters very
much where the property is stored; it's the locking that is the key
thing.  In other words, I think this *would* be possible if the
partition bound is stored as a property of individual partitions, as
long as it can't change without a lock on the parent.

However, it seems a lot better to make it a property of the parent
from a performance point of view.  Suppose there are 1000 partitions.
Reading one toasted value for pg_class and running stringToNode() on
it is probably a lot faster than scanning pg_inherits to find all of
the child partitions and then doing an index scan to find the pg_class
tuple for each and then decoding all of those tuples and assembling
them into some data structure.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 2016/08/31 16:17, Robert Haas wrote:
> On Wed, Aug 31, 2016 at 12:37 PM, Amit Langote wrote:
>> What I was trying to understand is why this would not be possible
>> with a design where partition bound is stored in the catalog as a property
>> of individual partitions instead of a design where we store collection of
>> partition bounds as a property of the parent.
> 
> From the point of view of feasibility, I don't think it matters very
> much where the property is stored; it's the locking that is the key
> thing.  In other words, I think this *would* be possible if the
> partition bound is stored as a property of individual partitions, as
> long as it can't change without a lock on the parent.
> 
> However, it seems a lot better to make it a property of the parent
> from a performance point of view.  Suppose there are 1000 partitions.
> Reading one toasted value for pg_class and running stringToNode() on
> it is probably a lot faster than scanning pg_inherits to find all of
> the child partitions and then doing an index scan to find the pg_class
> tuple for each and then decoding all of those tuples and assembling
> them into some data structure.

Seems worth trying.  One point that bothers me a bit is how do we enforce
partition bound condition on individual partition basis.  For example when
a row is inserted into a partition directly, we better check that it does
not fall outside the bounds and issue an error otherwise.  With current
approach, we just look up a partition's bound from the catalog and gin up
a check constraint expression (and cache in relcache) to be enforced in
ExecConstraints().  With the new approach, I guess we would need to look
up the parent's partition descriptor.  Note that the checking in
ExecConstraints() is turned off when routing a tuple from the parent.

Thanks,
Amit

On 2016/08/25 16:15, Ashutosh Bapat wrote:
> On Thu, Aug 25, 2016 at 12:22 PM, Amit Langote wrote:
>> b)
>> when accumulating append subpaths, do not flatten a subpath that is itself
>> an append when ((AppendPath *) subpath)->path.parent is a RelOptInfo with
>> non-NULL partitioning info.Is the latter somehow necessary for
>> pairwise-join considerations?
> 
> I don't think you need to do anything in the path creation code for this.
> As is it flattens all AppendPath hierarchies whether for partitioning or
> inheritance or subqueries. We should leave it as it is.

I thought it would be convenient for pairwise join code to work with the
hierarchy intact even within the AppendPath tree.  If it turns out to be
so, maybe that patch can take care of it.

>> I think I can manage to squeeze in (a) in the next version patch and will
>> also start working on (b), mainly the part about RelOptInfo getting some
>> partitioning info.
> 
> I am fine with b, where you would include some partitioning information in
> RelOptInfo. But you don't need to do what you said in (b) above.
> 
> In a private conversation Robert Haas suggested a way slightly different
> than what my patch for partition-wise join does. He suggested that the
> partitioning schemes i.e strategy, number of partitions and bounds of the
> partitioned elations involved in the query should be stored in PlannerInfo
> in the form of a list. Each partitioning scheme is annotated with the
> relids of the partitioned relations. RelOptInfo of the partitioned relation
> will point to the partitioning scheme in PlannerInfo. Along-with that each
> RelOptInfo will need to store partition keys for corresponding relation.
> This simplifies matching the partitioning schemes of the joining relations.
> Also it reduces the number of copies of partition bounds floating around as
> we expect that a query will involve multiple partitioned tables following
> similar partitioning schemes. May be you want to consider this idea while
> working on (b).

So IIUC, a partitioned relation's (baserel or joinrel) RelOptInfo has only
the information about partition keys.  They will be matched with query
restriction quals pruning away any unneeded partitions which happens
individually for each such parent baserel (within set_append_rel_size() I
suppose).  Further, two joining relations are eligible to be considered
for pairwise joining if they have identical partition keys and query
equi-join quals match the same.  The resulting joinrel will have the same
partition key (as either joining relation) and will have as many
partitions as there are in the intersection of sets of partitions of
joining rels (intersection proceeds by matching partition bounds).

"Partition scheme" structs go into a PlannerInfo list member, one
corresponding to each partitioned relation - baserel or joinrel, right?
As you say, each such struct has the following pieces of information:
strategy, num_partitions, bounds (and other auxiliary info).  Before
make_one_rel() starts, the list has one for each partitioned baserel.
After make_one_rel() has formed baserel pathlists and before
make_rel_from_joinlist() is called, are the partition scheme structs of
processed baserels marked with some information about the pruning activity
that occurred so far?  Then as we build successively higher levels of
joinrels, new entries will be made for those joinrels for which we added
pairwise join paths, with relids matching the corresponding joinrels.
Does that make sense?

Thanks,
Amit

On 2016/09/02 14:38, Ashutosh Bapat wrote:
> Here's something I observed with your set of patches posted in June. I have
> not checked the latest set of patches. So, if it's something fixed, please
> ignore the mail and sorry for me being lazy.
> 
> prt1 is partitioned table and it shows following information with \d+
> 
> regression=# \d+ prt1
>                         Partitioned table "public.prt1"
>  Column |       Type        | Modifiers | Storage  | Stats target |
> Description
> --------+-------------------+-----------+----------+--------------+-------------
>  a      | integer           |           | plain    |              |
>  b      | integer           |           | plain    |              |
>  c      | character varying |           | extended |              |
> Partition Key: PARTITION BY RANGE (a)
> Indexes:
>     "iprt1_a" btree (a)
> 
> Shouldn't we show all the partitions of this table and may be their ranges
> of lists?

Something I thought about as well.  I will implement that.

> I found the partitions from EXPLAIN plan
> 
> regression=# explain verbose select * from prt1;
>                               QUERY PLAN
> -----------------------------------------------------------------------
>  Append  (cost=0.00..6.00 rows=301 width=13)
>    ->  Seq Scan on public.prt1  (cost=0.00..0.00 rows=1 width=40)
>          Output: prt1.a, prt1.b, prt1.c
>    ->  Seq Scan on public.prt1_p1  (cost=0.00..2.25 rows=125 width=13)
>          Output: prt1_p1.a, prt1_p1.b, prt1_p1.c
>    ->  Seq Scan on public.prt1_p3  (cost=0.00..1.50 rows=50 width=13)
>          Output: prt1_p3.a, prt1_p3.b, prt1_p3.c
>    ->  Seq Scan on public.prt1_p2  (cost=0.00..2.25 rows=125 width=13)
>          Output: prt1_p2.a, prt1_p2.b, prt1_p2.c
> (9 rows)
> 
> Then did \d+ on each of those to find their ranges

[ ... ]

> 
> As you will observe that the table prt1 can not have any row with a < 0 and
> a > 600. But when I execute
> 
> regression=# explain verbose select * from prt1 where a > 1000000;
>                             QUERY PLAN
> ------------------------------------------------------------------
>  Append  (cost=0.00..0.00 rows=1 width=40)
>    ->  Seq Scan on public.prt1  (cost=0.00..0.00 rows=1 width=40)
>          Output: prt1.a, prt1.b, prt1.c
>          Filter: (prt1.a > 1000000)
> (4 rows)
> 
> it correctly excluded all the partitions, but did not exclude the parent
> relation. I guess, we have enough information to exclude it. Probably, we
> should add a check constraint on the parent which is OR of the check
> constraints on all the partitions. So there are two problems here
> 
> 1. \d+ doesn't show partitions - this is probably reported earlier, I don't
> remember.

You just did, :)

As I said I will implement that on lines of how inheritance children are
listed (with additional information ie, range or list).

> 2. A combination of constraints on the partitions should be applicable to
> the parent. We aren't doing that.

How about on seeing that a RELOPT_OTHER_MEMBER_REL is partitioned parent
table, we can have get_relation_constraints() include a constant false
clause in the list of constraints returned for
relation_excluded_by_constraints() to process so that it is not included
in the append result by way of constraint exclusion.  One more option is
to mark such rels dummy in set_rel_size().

Thanks,
Amit

<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px#ccc solid;padding-left:1ex"><span class=""><br /> > 2. A combination of constraints on the
partitionsshould be applicable to<br /> > the parent. We aren't doing that.<br /><br /></span>How about on seeing
thata RELOPT_OTHER_MEMBER_REL is partitioned parent<br /> table, we can have get_relation_constraints() include a
constantfalse<br /> clause in the list of constraints returned for<br /> relation_excluded_by_<wbr />constraints() to
processso that it is not included<br /> in the append result by way of constraint exclusion.  One more option is<br />
tomark such rels dummy in set_rel_size().<br /><br clear="all" /></blockquote></div><br /></div><div
class="gmail_extra">Iam not complaining about having parent relation there. For the people who are used to seeing the
parentrelation in the list of append relations, it may be awkward. But +1 if we can do that. If we can't do that, we
shouldat least mark with an OR of all constraints on the partitions, so that constraint exclusion can exclude it if
thereare conditions incompatible with constraints. This is what would happen in inheritance case as well, if there are
constraintson the parent. In the above example, the parent table would have constraints CHECK ((a >= 0 AND a <
250)OR (a >= 250 and a < 500) OR (a >= 500 or a < 600)). It will probably get excluded, if constraint
exclusionis smart enough to understand ORing.<br /></div><div class="gmail_extra">-- <br /><div class="gmail_signature"
data-smartmail="gmail_signature"><divdir="ltr">Best Wishes,<br />Ashutosh Bapat<br />EnterpriseDB Corporation<br />The
PostgresDatabase Company<br /></div></div></div></div> 

On 2016/09/02 15:22, Ashutosh Bapat wrote:
>>
>>
>>> 2. A combination of constraints on the partitions should be applicable to
>>> the parent. We aren't doing that.
>>
>> How about on seeing that a RELOPT_OTHER_MEMBER_REL is partitioned parent
>> table, we can have get_relation_constraints() include a constant false
>> clause in the list of constraints returned for
>> relation_excluded_by_constraints() to process so that it is not included
>> in the append result by way of constraint exclusion.  One more option is
>> to mark such rels dummy in set_rel_size().
>>
>>
> I am not complaining about having parent relation there. For the people who
> are used to seeing the parent relation in the list of append relations, it
> may be awkward. But +1 if we can do that. If we can't do that, we should at
> least mark with an OR of all constraints on the partitions, so that
> constraint exclusion can exclude it if there are conditions incompatible
> with constraints. This is what would happen in inheritance case as well, if
> there are constraints on the parent. In the above example, the parent table
> would have constraints CHECK ((a >= 0 AND a < 250) OR (a >= 250 and a <
> 500) OR (a >= 500 or a < 600)). It will probably get excluded, if
> constraint exclusion is smart enough to understand ORing.

I guess constraint exclusion would be (is) smart enough to handle that
correctly but why make it unnecessarily spend a *significant* amount of
time on doing the proof (when we *know* we can just skip it).  Imagine how
long the OR list could get.  By the way, my suggestion of just returning a
constant false clause also does not work - neither in case of a query with
restrictions (predicate has to be an OpExpr to go ahead with the proof
which constant false clause is not) nor in case of a query without
restrictions (proof is not performed at all).  So, that one is useless.

Getting rid of the parent table in the append list by other means may be a
way to go.  We know that the table is empty and safe to just drop.

Thanks,
Amit

On 2016/09/02 15:57, Ashutosh Bapat wrote:
> On Fri, Sep 2, 2016 at 12:23 PM, Amit Langote wrote:
>> Getting rid of the parent table in the append list by other means may be a
>> way to go.  We know that the table is empty and safe to just drop.
>>
> Ok. Does a constraint (1 = 2) or something like that which is obviously
> false, help?

No, anything like that would get reduced to a constant false clause by
eval_const_expressions() processing.

Thanks,
Amit

Hi,

On 2016/09/06 16:13, Ashutosh Bapat wrote:
> I found a server crash while running make check in regress folder. with
> this set of patches. Problem is RelationBuildPartitionKey() partexprsrc may
> be used uninitialized. Initializing it with NIL fixes the crash. Here's
> patch to fix it. Came up with the fix after discussion with Amit.

Thanks for the report.  Here is a rebased version of the patches including
you fix (no significant changes from those posted on Aug 26).

Thanks,
Amit

On Wed, Aug 31, 2016 at 1:05 PM, Amit Langote
<Langote_Amit_f8@lab.ntt.co.jp> wrote:
>> However, it seems a lot better to make it a property of the parent
>> from a performance point of view.  Suppose there are 1000 partitions.
>> Reading one toasted value for pg_class and running stringToNode() on
>> it is probably a lot faster than scanning pg_inherits to find all of
>> the child partitions and then doing an index scan to find the pg_class
>> tuple for each and then decoding all of those tuples and assembling
>> them into some data structure.
>
> Seems worth trying.  One point that bothers me a bit is how do we enforce
> partition bound condition on individual partition basis.  For example when
> a row is inserted into a partition directly, we better check that it does
> not fall outside the bounds and issue an error otherwise.  With current
> approach, we just look up a partition's bound from the catalog and gin up
> a check constraint expression (and cache in relcache) to be enforced in
> ExecConstraints().  With the new approach, I guess we would need to look
> up the parent's partition descriptor.  Note that the checking in
> ExecConstraints() is turned off when routing a tuple from the parent.

[ Sorry for the slow response. ]

Yeah, that's a problem.  Maybe it's best to associate this data with
the childrels after all - or halfway in between, e.g. augment
pg_inherits with this information.  After all, the performance problem
I was worried about above isn't really much of an issue: each backend
will build a relcache entry for the parent just once and then use it
for the lifetime of the session unless some invalidation occurs.  So
if that takes a small amount of extra time, it's probably not really a
big deal.  On the other hand, if we can't build the implicit
constraint for the child table without opening the parent, that's
probably going to cause us some serious inconvenience.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Hi,

On Tue, Sep 6, 2016 at 8:15 PM, Rajkumar Raghuwanshi
<rajkumar.raghuwanshi@enterprisedb.com> wrote:
> Hi,
>
> I have applied updated patches given by you, and observe below.
>
> here in the given example, t6_p3 partition is not allowed to have null, but
> I am able to insert it, causing two nulls in the table.
>
> --create a partition table
> create table t6 (a int, b varchar) partition by list(a);
> create table t6_p1 partition of t6 for values in (1,2,null);
> create table t6_p2 partition of t6 for values in (4,5);
> create table t6_p3 partition of t6 for values in (3,6);
>
> --insert some values
> insert into t6 select i,i::varchar from generate_series(1,6) i;
> insert into t6 values (null,'A');
>
> --try inserting null to t6_p3 partition table
> insert into t6_p3 values (null,'A');
>
> select tableoid::regclass,* from t6;
>  tableoid | a | b
> ----------+---+---
>  t6_p1    | 1 | 1
>  t6_p1    | 2 | 2
>  t6_p1    |   | A
>  t6_p2    | 4 | 4
>  t6_p2    | 5 | 5
>  t6_p3    | 3 | 3
>  t6_p3    | 6 | 6
>  t6_p3    |   | A
> (8 rows)

Thanks for testing!

That looks like a bug.  The same won't occur if you had inserted
through the parent (it would be correctly routed to the partition that
has been defined to accept nulls (that is, t6_p1 of your example). The
bug seems to have to do with the handling of nulls in generating
implicit constraints from the list of values of a given list
partition.  The direct insert on t6_p1 should have failed because
partition key has a value (null) violating the partition boundary
condition.  Will fix.

Thanks,
Amit

On Tue, Sep 6, 2016 at 9:19 PM, Robert Haas <robertmhaas@gmail.com> wrote:
> On Wed, Aug 31, 2016 at 1:05 PM, Amit Langote
> <Langote_Amit_f8@lab.ntt.co.jp> wrote:
>>> However, it seems a lot better to make it a property of the parent
>>> from a performance point of view.  Suppose there are 1000 partitions.
>>> Reading one toasted value for pg_class and running stringToNode() on
>>> it is probably a lot faster than scanning pg_inherits to find all of
>>> the child partitions and then doing an index scan to find the pg_class
>>> tuple for each and then decoding all of those tuples and assembling
>>> them into some data structure.
>>
>> Seems worth trying.  One point that bothers me a bit is how do we enforce
>> partition bound condition on individual partition basis.  For example when
>> a row is inserted into a partition directly, we better check that it does
>> not fall outside the bounds and issue an error otherwise.  With current
>> approach, we just look up a partition's bound from the catalog and gin up
>> a check constraint expression (and cache in relcache) to be enforced in
>> ExecConstraints().  With the new approach, I guess we would need to look
>> up the parent's partition descriptor.  Note that the checking in
>> ExecConstraints() is turned off when routing a tuple from the parent.
>
> [ Sorry for the slow response. ]
>
> Yeah, that's a problem.  Maybe it's best to associate this data with
> the childrels after all - or halfway in between, e.g. augment
> pg_inherits with this information.  After all, the performance problem
> I was worried about above isn't really much of an issue: each backend
> will build a relcache entry for the parent just once and then use it
> for the lifetime of the session unless some invalidation occurs.  So
> if that takes a small amount of extra time, it's probably not really a
> big deal.  On the other hand, if we can't build the implicit
> constraint for the child table without opening the parent, that's
> probably going to cause us some serious inconvenience.

Agreed.  So I will stick with the existing approach.

Thanks,
Amit

Hi,

On 2016/09/07 17:56, Rajkumar Raghuwanshi wrote:
> Hi,
> 
> I have a query regarding list partitioning,
> 
> For example if I want to store employee data in a table, with "IT" dept
> employee in emp_p1 partition, "HR" dept employee in emp_p2 partition and if
> employee belongs to other than these two, should come in emp_p3 partition.
> 
> In this case not sure how to create partition table. Do we have something
> like we have UNBOUNDED for range partition or oracle have "DEFAULT" for
> list partition.
> 
> create table employee (empid int, dept varchar) partition by list(dept);
> create table emp_p1 partition of employee for values in ('IT');
> create table emp_p2 partition of employee for values in ('HR');
> create table emp_p3 partition of employee for values in (??);

Sorry, no such feature is currently offered.  It might be possible to
offer something like a "default" list partition which accepts values other
than those specified for other existing partitions.  However, that means
if we add a non-default list partition after a default one has been
created, the implementation must make sure that it moves any values from
the default partition that now belong to the newly created partition.

Thanks,
Amit

On 2016/09/08 21:38, Rajkumar Raghuwanshi wrote:
> On Wed, Sep 7, 2016 at 3:58 PM, Amit Langote wrote:
>> On 2016/09/07 17:56, Rajkumar Raghuwanshi wrote:
>>>
>>> In this case not sure how to create partition table. Do we have something
>>> like we have UNBOUNDED for range partition or oracle have "DEFAULT" for
>>> list partition.
>>>
>>> create table employee (empid int, dept varchar) partition by list(dept);
>>> create table emp_p1 partition of employee for values in ('IT');
>>> create table emp_p2 partition of employee for values in ('HR');
>>> create table emp_p3 partition of employee for values in (??);
>>
>> Sorry, no such feature is currently offered.  It might be possible to
>> offer something like a "default" list partition which accepts values other
>> than those specified for other existing partitions.  However, that means
>> if we add a non-default list partition after a default one has been
>> created, the implementation must make sure that it moves any values from
>> the default partition that now belong to the newly created partition.
> 
> Thanks for clarifying, But I could see same problem of moving data when
> adding a new non-default partition with unbounded range partition.
> 
> For example give here, Initially I have create a partition table test with
> test_p3 as unbounded end,
> Later tried to change test_p3 to contain 7-9 values only, and adding a new
> partition test_p4 contain 10-unbound.
> 
> --create partition table and some leafs
> CREATE TABLE test (a int, b int) PARTITION BY RANGE(a);
> CREATE TABLE test_p1 PARTITION OF test FOR VALUES START (1) END (4);
> CREATE TABLE test_p2 PARTITION OF test FOR VALUES START (4) END (7);
> CREATE TABLE test_p3 PARTITION OF test FOR VALUES START (7) END UNBOUNDED;
> 
> --insert some data
> INSERT INTO test SELECT i, i*10 FROM generate_series(1,3) i;
> INSERT INTO test SELECT i, i*10 FROM generate_series(4,6) i;
> INSERT INTO test SELECT i, i*10 FROM generate_series(7,13) i;
> 
> --directly not able to attach test_p4 because of overlap error, hence
> detached test_p3 and than attaching test_p4
> SELECT tableoid::regclass,* FROM test;
>  tableoid | a  |  b
> ----------+----+-----
>  test_p1  |  1 |  10
>  test_p1  |  2 |  20
>  test_p1  |  3 |  30
>  test_p2  |  4 |  40
>  test_p2  |  5 |  50
>  test_p2  |  6 |  60
>  test_p3  |  7 |  70
>  test_p3  |  8 |  80
>  test_p3  |  9 |  90
>  test_p3  | 10 | 100
>  test_p3  | 11 | 110
>  test_p3  | 12 | 120
>  test_p3  | 13 | 130
> (13 rows)
> 
> ALTER TABLE test DETACH PARTITION test_p3;
> CREATE TABLE test_p4 (like test);
> ALTER TABLE test ATTACH PARTITION test_p4 FOR VALUES start (10) end
> UNBOUNDED;
> 
> --now can not attach test_p3 because of overlap with test_p4, causing data
> loss from main test table.
> ALTER TABLE test ATTACH PARTITION test_p3 FOR VALUES start (7) end (10);
> ERROR:  source table contains a row violating partition bound specification
> ALTER TABLE test ATTACH PARTITION test_p3 FOR VALUES start (7) end (13);
> ERROR:  partition "test_p3" would overlap partition "test_p4"

In this particular case, you will have to move any rows in test_p3 with
key > or >= 10 into the new partition test_p4 using dml (to not lose any
data).  Then attach test_p3 as partition for values start (7) end (10);
you won't get either of the above errors.

Looking forward, what we need I think is a split partition command.
Adding a new partition that overlaps the default list partition or
unbounded range partition could be done by splitting the latter. Perhaps
something like:

alter table test split partition test_p3 at (10) [inclusive | exclusive] with test_p4;

The above command would make test_p3 into 2 partitions:
 test_p3 start (7) end (10) [inclusive | exclusive] test_p4 start (10) [exclusive | inclusive] end unbounded

Any rows in test_p3 with key > or >= 10 will be moved into the newly
created test_p4 as part of the execution of this command.

For your list partitioning example:

create table employee (empid int, dept varchar) partition by list(dept);
create table emp_p1 partition of employee for values in ('IT');
create table emp_p2 partition of employee for values in ('HR');
create table emp_p3 partition of employee for values in (default);

alter table emp split partition emp_p3 with emp_p3 ('ACCT') emp_p4 (default);

Any rows in emp_p3 with key != 'ACCT' will be moved into the newly created
default partition emp_p4.


But for time being, I think we could provide the syntax and mechanism for
default list partition seeing as we have the same for range partitioned
table (namely a range partition with unbounded start or end).  Although
with the limitations as discussed.

Thoughts?

Thanks,
Amit

On 2016/09/06 22:04, Amit Langote wrote:
> Will fix.

Here is an updated set of patches.

In addition to fixing a couple of bugs reported by Ashutosh and Rajkumar,
there are a few of major changes:

* change the way individual partition bounds are represented internally
  and the way a collection of partition bounds associated with a
  partitioned table is exposed to other modules.  Especially list
  partition bounds which are manipulated more efficiently as discussed
  at [1].

* \d partitioned_table now shows partition count and \d+ lists partition
  names and their bounds as follows:

\d t6
           Table "public.t6"
 Column |       Type        | Modifiers
.-------+-------------------+-----------
 a      | integer           |
 b      | character varying |
Partition Key: LIST (a)
Number of partitions: 3 (Use \d+ to list them.)

\d+ t6
                               Table "public.t6"
 Column |       Type        | Modifiers | Storage  | Stats target |
Description
.-------+-------------------+-----------+----------+--------------+-------------
 a      | integer           |           | plain    |              |
 b      | character varying |           | extended |              |
Partition Key: LIST (a)
Partitions: t6_p1 FOR VALUES IN (1, 2, NULL),
            t6_p2 FOR VALUES IN (4, 5),
            t6_p3 FOR VALUES IN (3, 6)

\d+ p
                             Table "public.p"
 Column |     Type     | Modifiers | Storage  | Stats target | Description
.-------+--------------+-----------+----------+--------------+-------------
 a      | integer      |           | plain    |              |
 b      | character(1) |           | extended |              |
Partition Key: RANGE (a)
Partitions: p1 FOR VALUES START (1) END (10),
            p2 FOR VALUES START (10) END (20),
            p3 FOR VALUES START (20) END (30),
            p4 FOR VALUES START (30) EXCLUSIVE END (40) INCLUSIVE,
            p5 FOR VALUES START (40) EXCLUSIVE END (50),
            p6 FOR VALUES START (50) END UNBOUNDED

* Some more regression tests

Thanks,
Amit

[1]
https://www.postgresql.org/message-id/CA%2BTgmoZCr0-t93KgJA3T1uy9yWxfYaSYL3X35ObyHg%2BZUfERqQ%40mail.gmail.com

Hi

On 2016/09/09 17:55, Amit Langote wrote:
> On 2016/09/06 22:04, Amit Langote wrote:
>> Will fix.
>
> Here is an updated set of patches.

An email from Rajkumar somehow managed to break out of this thread.
Quoting his message below so that I don't end up replying with patches on
two different threads.

On 2016/09/14 16:58, Rajkumar Raghuwanshi wrote:
> I have Continued with testing declarative partitioning with the latest
> patch. Got some more observation, given below

Thanks a lot for testing.

> -- Observation 1 : Getting overlap error with START with EXCLUSIVE in range
> partition.
>
> create table test_range_bound ( a int) partition by range(a);
> --creating a partition to contain records {1,2,3,4}, by default 1 is
> inclusive and 5 is exclusive
> create table test_range_bound_p1 partition of test_range_bound for values
> start (1) end (5);
> --now trying to create a partition by explicitly mentioning start is
> exclusive to contain records {5,6,7}, here trying to create with START with
> 4 as exclusive so range should be 5 to 8, but getting partition overlap
> error.
> create table test_range_bound_p2 partition of test_range_bound for values
> start (4) EXCLUSIVE end (8);
> ERROR:  partition "test_range_bound_p2" would overlap partition
> "test_range_bound_p1"

Wow, this is bad.  What is needed in this case is "canonicalization" of
the range partition bounds specified in the command.  Range types do this
and hence an equivalent test done with range type values would disagree
with the result given by the patch for range partition bounds.

select '[1,5)'::int4range && '(4,8]'::int4range as cmp;
 cmp
-----
 f
(1 row)

In this case, the second range is converted into its equivalent canonical
form viz. '[5, 9)'.  Then comparison of bounds 5) and [5 can tell that the
ranges do not overlap after all.  Range type operators can do this because
their code can rely on the availability of a canonicalization function for
a given range type.  From the range types documentation:

"""
A discrete range type should have a canonicalization function that is
aware of the desired step size for the element type. The canonicalization
function is charged with converting equivalent values of the range type to
have identical representations, in particular consistently inclusive or
exclusive bounds. If a canonicalization function is not specified, then
ranges with different formatting will always be treated as unequal, even
though they might represent the same set of values in reality.
"""

to extend the last sentence:

"... or consider two ranges overlapping when in reality they are not
(maybe they are really just adjacent)."

Within the code handling range partition bound, no such canonicalization
happens, so comparison 5) and (4 ends up concluding that upper1 > lower2,
hence ranges overlap.

To mitigate this, how about we restrict range partition key to contain
columns of only those types for which we know we can safely canonicalize a
range bound (ie, discrete range types)?  I don't think we can use, say,
existing int4range_canonical but will have to write a version of it for
partitioning usage (range bounds of partitions are different from what
int4range_canonical is ready to handle).  This approach will be very
limiting as then range partitions will be limited to columns of int,
bigint and date type only.

One more option is we let the user specify the canonicalize function next
to the column name when defining the partition key.  If not specified, we
hard-code one for the types for which we will be implementing a
canonicalize function (ie, above mentioned types).  In other cases, we
just don't have one and hence if an unexpected result occurs when creating
a new partition, it's up to the user to realize what happened.  Of course,
we will be mentioning in the documentation why a canonicalize function is
necessary and how to write one.  Note that this canonicalize function
comes into play only when defining new partitions, it has no role beyond
that point.

> -- Observation 2 : able to create sub-partition out of the range set for
> main table, causing not able to insert data satisfying any of the partition.
>
> create table test_subpart (c1 int) partition by range (c1);
> create table test_subpart_p1 partition of test_subpart for values start (1)
> end (100) inclusive partition by range (c1);
> create table test_subpart_p1_sub1 partition of test_subpart_p1 for values
> start (101) end (200);
>
> \d+ test_subpart
>                      Table "public.test_subpart"
>  Column |  Type   | Modifiers | Storage | Stats target | Description
> --------+---------+-----------+---------+--------------+-------------
>  c1     | integer |           | plain   |              |
> Partition Key: RANGE (c1)
> Partitions: test_subpart_p1 FOR VALUES START (1) END (100) INCLUSIVE
>
> \d+ test_subpart_p1
>                    Table "public.test_subpart_p1"
>  Column |  Type   | Modifiers | Storage | Stats target | Description
> --------+---------+-----------+---------+--------------+-------------
>  c1     | integer |           | plain   |              |
> Partition Of: test_subpart FOR VALUES START (1) END (100) INCLUSIVE
> Partition Key: RANGE (c1)
> Partitions: test_subpart_p1_sub1 FOR VALUES START (101) END (200)
>
> insert into test_subpart values (50);
> ERROR:  no partition of relation "test_subpart_p1" found for row
> DETAIL:  Failing row contains (50).
> insert into test_subpart values (150);
> ERROR:  no partition of relation "test_subpart" found for row
> DETAIL:  Failing row contains (150).

It seems that DDL should prevent the same column being used in partition
key of lower level partitions.  I don't know how much sense it would make,
but being able to use the same column as partition key of lower level
partitions may be a feature useful to some users if they know what they
are doing.  But this last part doesn't sound like a good thing.  I
modified the patch such that lower level partitions cannot use columns
used by ancestor tables.

> -- Observation 3 : Getting cache lookup failed, when selecting list
> partition table containing array.
>
> CREATE TABLE test_array ( i int,j int[],k text[]) PARTITION BY LIST (j);
> CREATE TABLE test_array_p1 PARTITION OF test_array FOR VALUES IN ('{1}');
> CREATE TABLE test_array_p2 PARTITION OF test_array FOR VALUES IN ('{2,2}');
>
> INSERT INTO test_array (i,j[1],k[1]) VALUES (1,1,1);
> INSERT INTO test_array (i,j[1],j[2],k[1]) VALUES (2,2,2,2);
>
> postgres=# SELECT tableoid::regclass,* FROM test_array_p1;
>    tableoid    | i |  j  |  k
> ---------------+---+-----+-----
>  test_array_p1 | 1 | {1} | {1}
> (1 row)
>
> postgres=# SELECT tableoid::regclass,* FROM test_array_p2;
>    tableoid    | i |   j   |  k
> ---------------+---+-------+-----
>  test_array_p2 | 2 | {2,2} | {2}
> (1 row)
>
> postgres=# SELECT tableoid::regclass,* FROM test_array;
> ERROR:  cache lookup failed for type 0

That's a bug.  Fixed in the attached patch.

PS: I'm going to have limited Internet access during this weekend and over
the next week, so responses could be slow.  Sorry about that.

Thanks,
Amit

Hi Amit,

It looks like there is some problem while creating paramterized paths
for multi-level partitioned tables. Here's a longish testcase

CREATE TABLE prt1_l (a int, b int, c varchar) PARTITION BY RANGE(a);
CREATE TABLE prt1_l_p1 PARTITION OF prt1_l FOR VALUES START (0) END
(250) PARTITION BY RANGE (b);
CREATE TABLE prt1_l_p1_p1 PARTITION OF prt1_l_p1 FOR VALUES START (0) END (100);
CREATE TABLE prt1_l_p1_p2 PARTITION OF prt1_l_p1 FOR VALUES START
(100) END (250);
CREATE TABLE prt1_l_p2 PARTITION OF prt1_l FOR VALUES START (250) END
(500) PARTITION BY RANGE (c);
CREATE TABLE prt1_l_p2_p1 PARTITION OF prt1_l_p2 FOR VALUES START
('0250') END ('0400');
CREATE TABLE prt1_l_p2_p2 PARTITION OF prt1_l_p2 FOR VALUES START
('0400') END ('0500');
CREATE TABLE prt1_l_p3 PARTITION OF prt1_l FOR VALUES START (500) END
(600) PARTITION BY RANGE ((b + a));
CREATE TABLE prt1_l_p3_p1 PARTITION OF prt1_l_p3 FOR VALUES START
(1000) END (1100);
CREATE TABLE prt1_l_p3_p2 PARTITION OF prt1_l_p3 FOR VALUES START
(1100) END (1200);
INSERT INTO prt1_l SELECT i, i, to_char(i, 'FM0000') FROM
generate_series(0, 599, 2) i;
CREATE TABLE uprt1_l AS SELECT * FROM prt1_l;

CREATE TABLE prt2_l (a int, b int, c varchar) PARTITION BY RANGE(b);
CREATE TABLE prt2_l_p1 PARTITION OF prt2_l FOR VALUES START (0) END
(250) PARTITION BY RANGE (a);
CREATE TABLE prt2_l_p1_p1 PARTITION OF prt2_l_p1 FOR VALUES START (0) END (100);
CREATE TABLE prt2_l_p1_p2 PARTITION OF prt2_l_p1 FOR VALUES START
(100) END (250);
CREATE TABLE prt2_l_p2 PARTITION OF prt2_l FOR VALUES START (250) END
(500) PARTITION BY RANGE (c);
CREATE TABLE prt2_l_p2_p1 PARTITION OF prt2_l_p2 FOR VALUES START
('0250') END ('0400');
CREATE TABLE prt2_l_p2_p2 PARTITION OF prt2_l_p2 FOR VALUES START
('0400') END ('0500');
CREATE TABLE prt2_l_p3 PARTITION OF prt2_l FOR VALUES START (500) END
(600) PARTITION BY RANGE ((a + b));
CREATE TABLE prt2_l_p3_p1 PARTITION OF prt2_l_p3 FOR VALUES START
(1000) END (1100);
CREATE TABLE prt2_l_p3_p2 PARTITION OF prt2_l_p3 FOR VALUES START
(1100) END (1200);
INSERT INTO prt2_l SELECT i, i, to_char(i, 'FM0000') FROM
generate_series(0, 599, 3) i;
CREATE TABLE uprt2_l AS SELECT * FROM prt2_l;

EXPLAIN (VERBOSE, COSTS OFF)
SELECT * FROM prt1_l t1 LEFT JOIN LATERAL             (SELECT t2.a AS t2a, t2.c AS t2c, t2.b AS t2b, t3.a AS
t3a, least(t1.a,t2.a,t3.a) FROM prt1_l t2 JOIN prt2_l t3 ON (t2.a =
t3.b AND t2.b = t3.a AND t2.c = t3.c AND t2.b + t2.a = t3.a + t3.b))
ss             ON t1.a = ss.t2a AND t1.b = ss.t2a AND t1.c = ss.t2c AND
t1.b + t1.a = ss.t2a + ss.t2b WHERE t1.a % 25 = 0 ORDER BY t1.a;
ERROR:  could not devise a query plan for the given query

Let's replace the laterally referenced relation by an unpartitioned table.

EXPLAIN (VERBOSE, COSTS OFF)
SELECT * FROM uprt1_l t1 LEFT JOIN LATERAL             (SELECT t2.a AS t2a, t2.c AS t2c, t2.b AS t2b, t3.a AS
t3a, least(t1.a,t2.a,t3.a) FROM prt1_l t2 JOIN prt2_l t3 ON (t2.a =
t3.b AND t2.b = t3.a AND t2.c = t3.c AND t2.b + t2.a = t3.a + t3.b))
ss             ON t1.a = ss.t2a AND t1.b = ss.t2a AND t1.c = ss.t2c AND
t1.b + t1.a = ss.t2a + ss.t2b WHERE t1.a % 25 = 0 ORDER BY t1.a;
ERROR:  could not devise a query plan for the given query

Let's replace another partitioned table in the inner query with an
unpartitioned table.

EXPLAIN (VERBOSE, COSTS OFF)
SELECT * FROM uprt1_l t1 LEFT JOIN LATERAL             (SELECT t2.a AS t2a, t2.c AS t2c, t2.b AS t2b, t3.a AS
t3a, least(t1.a,t2.a,t3.a) FROM prt1_l t2 JOIN uprt2_l t3 ON (t2.a =
t3.b AND t2.b = t3.a AND t2.c = t3.c AND t2.b + t2.a = t3.a + t3.b))
ss             ON t1.a = ss.t2a AND t1.b = ss.t2a AND t1.c = ss.t2c AND
t1.b + t1.a = ss.t2a + ss.t2b WHERE t1.a % 25 = 0 ORDER BY t1.a;
ERROR:  could not devise a query plan for the given query

Please check if you are able to reproduce these errors in your
repository. I made sure that I cleaned up all partition-wise join code
before testing this, but ... .

I tried to debug the problem somewhat. In set_append_rel_pathlist(),
it finds that at least one child has a parameterized path as the
cheapest path, so it doesn't create an unparameterized path for append
rel. At the same time there is a parameterization common to all the
children, so it doesn't create any path. There seem to be two problems
here
1. The children from second level onwards may not be getting
parameterized for lateral references. That seems unlikely but
possible.
2. Reparameterization should have corrected this, but
reparameterize_path() does not support AppendPaths.

On Thu, Sep 15, 2016 at 2:23 PM, Amit Langote
<Langote_Amit_f8@lab.ntt.co.jp> wrote:
>
> Hi
>
> On 2016/09/09 17:55, Amit Langote wrote:
>> On 2016/09/06 22:04, Amit Langote wrote:
>>> Will fix.
>>
>> Here is an updated set of patches.
>
> An email from Rajkumar somehow managed to break out of this thread.
> Quoting his message below so that I don't end up replying with patches on
> two different threads.
>
> On 2016/09/14 16:58, Rajkumar Raghuwanshi wrote:
>> I have Continued with testing declarative partitioning with the latest
>> patch. Got some more observation, given below
>
> Thanks a lot for testing.
>
>> -- Observation 1 : Getting overlap error with START with EXCLUSIVE in range
>> partition.
>>
>> create table test_range_bound ( a int) partition by range(a);
>> --creating a partition to contain records {1,2,3,4}, by default 1 is
>> inclusive and 5 is exclusive
>> create table test_range_bound_p1 partition of test_range_bound for values
>> start (1) end (5);
>> --now trying to create a partition by explicitly mentioning start is
>> exclusive to contain records {5,6,7}, here trying to create with START with
>> 4 as exclusive so range should be 5 to 8, but getting partition overlap
>> error.
>> create table test_range_bound_p2 partition of test_range_bound for values
>> start (4) EXCLUSIVE end (8);
>> ERROR:  partition "test_range_bound_p2" would overlap partition
>> "test_range_bound_p1"
>
> Wow, this is bad.  What is needed in this case is "canonicalization" of
> the range partition bounds specified in the command.  Range types do this
> and hence an equivalent test done with range type values would disagree
> with the result given by the patch for range partition bounds.
>
> select '[1,5)'::int4range && '(4,8]'::int4range as cmp;
>  cmp
> -----
>  f
> (1 row)
>
> In this case, the second range is converted into its equivalent canonical
> form viz. '[5, 9)'.  Then comparison of bounds 5) and [5 can tell that the
> ranges do not overlap after all.  Range type operators can do this because
> their code can rely on the availability of a canonicalization function for
> a given range type.  From the range types documentation:
>
> """
> A discrete range type should have a canonicalization function that is
> aware of the desired step size for the element type. The canonicalization
> function is charged with converting equivalent values of the range type to
> have identical representations, in particular consistently inclusive or
> exclusive bounds. If a canonicalization function is not specified, then
> ranges with different formatting will always be treated as unequal, even
> though they might represent the same set of values in reality.
> """
>
> to extend the last sentence:
>
> "... or consider two ranges overlapping when in reality they are not
> (maybe they are really just adjacent)."
>
> Within the code handling range partition bound, no such canonicalization
> happens, so comparison 5) and (4 ends up concluding that upper1 > lower2,
> hence ranges overlap.
>
> To mitigate this, how about we restrict range partition key to contain
> columns of only those types for which we know we can safely canonicalize a
> range bound (ie, discrete range types)?  I don't think we can use, say,
> existing int4range_canonical but will have to write a version of it for
> partitioning usage (range bounds of partitions are different from what
> int4range_canonical is ready to handle).  This approach will be very
> limiting as then range partitions will be limited to columns of int,
> bigint and date type only.
>
> One more option is we let the user specify the canonicalize function next
> to the column name when defining the partition key.  If not specified, we
> hard-code one for the types for which we will be implementing a
> canonicalize function (ie, above mentioned types).  In other cases, we
> just don't have one and hence if an unexpected result occurs when creating
> a new partition, it's up to the user to realize what happened.  Of course,
> we will be mentioning in the documentation why a canonicalize function is
> necessary and how to write one.  Note that this canonicalize function
> comes into play only when defining new partitions, it has no role beyond
> that point.
>
>> -- Observation 2 : able to create sub-partition out of the range set for
>> main table, causing not able to insert data satisfying any of the partition.
>>
>> create table test_subpart (c1 int) partition by range (c1);
>> create table test_subpart_p1 partition of test_subpart for values start (1)
>> end (100) inclusive partition by range (c1);
>> create table test_subpart_p1_sub1 partition of test_subpart_p1 for values
>> start (101) end (200);
>>
>> \d+ test_subpart
>>                      Table "public.test_subpart"
>>  Column |  Type   | Modifiers | Storage | Stats target | Description
>> --------+---------+-----------+---------+--------------+-------------
>>  c1     | integer |           | plain   |              |
>> Partition Key: RANGE (c1)
>> Partitions: test_subpart_p1 FOR VALUES START (1) END (100) INCLUSIVE
>>
>> \d+ test_subpart_p1
>>                    Table "public.test_subpart_p1"
>>  Column |  Type   | Modifiers | Storage | Stats target | Description
>> --------+---------+-----------+---------+--------------+-------------
>>  c1     | integer |           | plain   |              |
>> Partition Of: test_subpart FOR VALUES START (1) END (100) INCLUSIVE
>> Partition Key: RANGE (c1)
>> Partitions: test_subpart_p1_sub1 FOR VALUES START (101) END (200)
>>
>> insert into test_subpart values (50);
>> ERROR:  no partition of relation "test_subpart_p1" found for row
>> DETAIL:  Failing row contains (50).
>> insert into test_subpart values (150);
>> ERROR:  no partition of relation "test_subpart" found for row
>> DETAIL:  Failing row contains (150).
>
> It seems that DDL should prevent the same column being used in partition
> key of lower level partitions.  I don't know how much sense it would make,
> but being able to use the same column as partition key of lower level
> partitions may be a feature useful to some users if they know what they
> are doing.  But this last part doesn't sound like a good thing.  I
> modified the patch such that lower level partitions cannot use columns
> used by ancestor tables.
>
>> -- Observation 3 : Getting cache lookup failed, when selecting list
>> partition table containing array.
>>
>> CREATE TABLE test_array ( i int,j int[],k text[]) PARTITION BY LIST (j);
>> CREATE TABLE test_array_p1 PARTITION OF test_array FOR VALUES IN ('{1}');
>> CREATE TABLE test_array_p2 PARTITION OF test_array FOR VALUES IN ('{2,2}');
>>
>> INSERT INTO test_array (i,j[1],k[1]) VALUES (1,1,1);
>> INSERT INTO test_array (i,j[1],j[2],k[1]) VALUES (2,2,2,2);
>>
>> postgres=# SELECT tableoid::regclass,* FROM test_array_p1;
>>    tableoid    | i |  j  |  k
>> ---------------+---+-----+-----
>>  test_array_p1 | 1 | {1} | {1}
>> (1 row)
>>
>> postgres=# SELECT tableoid::regclass,* FROM test_array_p2;
>>    tableoid    | i |   j   |  k
>> ---------------+---+-------+-----
>>  test_array_p2 | 2 | {2,2} | {2}
>> (1 row)
>>
>> postgres=# SELECT tableoid::regclass,* FROM test_array;
>> ERROR:  cache lookup failed for type 0
>
> That's a bug.  Fixed in the attached patch.
>
> PS: I'm going to have limited Internet access during this weekend and over
> the next week, so responses could be slow.  Sorry about that.
>
> Thanks,
> Amit



-- 
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company

On Thu, Sep 15, 2016 at 4:53 AM, Amit Langote
<Langote_Amit_f8@lab.ntt.co.jp> wrote:
> Wow, this is bad.  What is needed in this case is "canonicalization" of
> the range partition bounds specified in the command.

I think we shouldn't worry about this.  It seems like unnecessary
scope creep.  All human beings capable of using PostgreSQL will
understand that there are no integers between 4 and 5, so any
practical impact on this would be on someone creating partitions
automatically.  But if someone is creating partitions automatically
they are highly likely to be using the same EXCLUSIVE/INCLUSIVE
settings for all of the partitions, in which case this won't arise.
And if they aren't, then I think we should just make them deal with
this limitation in their code instead of dealing with it in our code.
This patch is plenty complicated enough already; introducing a whole
new canonicalization concept that will help practically nobody seems
to me to be going in the wrong direction.  If somebody really cares
enough to want to try to fix this, they can submit a followup patch
someday.

> To mitigate this, how about we restrict range partition key to contain
> columns of only those types for which we know we can safely canonicalize a
> range bound (ie, discrete range types)?  I don't think we can use, say,
> existing int4range_canonical but will have to write a version of it for
> partitioning usage (range bounds of partitions are different from what
> int4range_canonical is ready to handle).  This approach will be very
> limiting as then range partitions will be limited to columns of int,
> bigint and date type only.

-1.  That is letting the tail wag the dog.  Let's leave it the way you
had it and be happy.

>> -- Observation 2 : able to create sub-partition out of the range set for
>> main table, causing not able to insert data satisfying any of the partition.
>>
>> create table test_subpart (c1 int) partition by range (c1);
>> create table test_subpart_p1 partition of test_subpart for values start (1)
>> end (100) inclusive partition by range (c1);
>> create table test_subpart_p1_sub1 partition of test_subpart_p1 for values
>> start (101) end (200);
>
> It seems that DDL should prevent the same column being used in partition
> key of lower level partitions.  I don't know how much sense it would make,
> but being able to use the same column as partition key of lower level
> partitions may be a feature useful to some users if they know what they
> are doing.  But this last part doesn't sound like a good thing.  I
> modified the patch such that lower level partitions cannot use columns
> used by ancestor tables.

I again disagree.  If somebody defines partition bounds that make it
impossible to insert the data that they care about, that's operator
error.  The fact that, across multiple levels, you can manage to make
it impossible to insert any data at all does not make it anything
other than operator error.  If we take the contrary position that it's
the system's job to prevent this sort of thing, we may disallow some
useful cases, like partitioning by the year portion of a date and then
subpartitioning by the month portion of that same date.

I think we'll probably also find that we're making the code
complicated to no purpose.  For example, now you have to check when
attaching a partition that it doesn't violate the rule; otherwise you
end up with a table that can't be created directly (and thus can't
survive dump-and-restore) but can be created indirectly by exploiting
loopholes in the checks.  It's tempting to think that we can check
simple cases - e.g. if the parent and the child are partitioning on
the same exact column, the child's range should be contained within
the parent's range - but more complicated cases are tricky.  Suppose
the table is range-partitioned on (a, b) and range-subpartitioned on
b.  It's not trivial to figure out whether the set of values that the
user can insert into that partition is non-empty.  If we allow
partitioning on expressions, then it quickly becomes altogether
impossible to deduce anything useful - unless you can solve the
halting problem.

And, really, why do we care?  If the user creates a partitioning
scheme that permits no rows in some or all of the partitions, then
they will have an empty table that can be correctly dumped and
restored but which cannot be used for anything useful unless it is
modified first.  They probably don't want that, but it's not any more
broken than a table inheritance setup with mutually exclusive CHECK
constraints, or for that matter a plain table with mutually exclusive
CHECK constraints - and we don't try to prohibit those things.  This
patch is supposed to be implementing partitioning, not artificial
intelligence.

>> -- Observation 3 : Getting cache lookup failed, when selecting list
>> partition table containing array.
>>
>> postgres=# SELECT tableoid::regclass,* FROM test_array;
>> ERROR:  cache lookup failed for type 0
>
> That's a bug.  Fixed in the attached patch.

Now on this one I'm not going to argue with your analysis.  :-)

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Thu, Sep 15, 2016 at 10:07 PM, Robert Haas <robertmhaas@gmail.com> wrote:
> On Thu, Sep 15, 2016 at 4:53 AM, Amit Langote
> <Langote_Amit_f8@lab.ntt.co.jp> wrote:
>> Wow, this is bad.  What is needed in this case is "canonicalization" of
>> the range partition bounds specified in the command.
>
> I think we shouldn't worry about this.  It seems like unnecessary
> scope creep.  All human beings capable of using PostgreSQL will
> understand that there are no integers between 4 and 5, so any
> practical impact on this would be on someone creating partitions
> automatically.  But if someone is creating partitions automatically
> they are highly likely to be using the same EXCLUSIVE/INCLUSIVE
> settings for all of the partitions, in which case this won't arise.
> And if they aren't, then I think we should just make them deal with
> this limitation in their code instead of dealing with it in our code.
> This patch is plenty complicated enough already; introducing a whole
> new canonicalization concept that will help practically nobody seems
> to me to be going in the wrong direction.  If somebody really cares
> enough to want to try to fix this, they can submit a followup patch
> someday.
>
>> To mitigate this, how about we restrict range partition key to contain
>> columns of only those types for which we know we can safely canonicalize a
>> range bound (ie, discrete range types)?  I don't think we can use, say,
>> existing int4range_canonical but will have to write a version of it for
>> partitioning usage (range bounds of partitions are different from what
>> int4range_canonical is ready to handle).  This approach will be very
>> limiting as then range partitions will be limited to columns of int,
>> bigint and date type only.
>
> -1.  That is letting the tail wag the dog.  Let's leave it the way you
> had it and be happy.

Alright, let's leave this as something to work out later.  We will
have to document the fact that such limitation exists though, I'd
think.

>>> -- Observation 2 : able to create sub-partition out of the range set for
>>> main table, causing not able to insert data satisfying any of the partition.
>>>
>>> create table test_subpart (c1 int) partition by range (c1);
>>> create table test_subpart_p1 partition of test_subpart for values start (1)
>>> end (100) inclusive partition by range (c1);
>>> create table test_subpart_p1_sub1 partition of test_subpart_p1 for values
>>> start (101) end (200);
>>
>> It seems that DDL should prevent the same column being used in partition
>> key of lower level partitions.  I don't know how much sense it would make,
>> but being able to use the same column as partition key of lower level
>> partitions may be a feature useful to some users if they know what they
>> are doing.  But this last part doesn't sound like a good thing.  I
>> modified the patch such that lower level partitions cannot use columns
>> used by ancestor tables.
>
> I again disagree.  If somebody defines partition bounds that make it
> impossible to insert the data that they care about, that's operator
> error.  The fact that, across multiple levels, you can manage to make
> it impossible to insert any data at all does not make it anything
> other than operator error.  If we take the contrary position that it's
> the system's job to prevent this sort of thing, we may disallow some
> useful cases, like partitioning by the year portion of a date and then
> subpartitioning by the month portion of that same date.
>
> I think we'll probably also find that we're making the code
> complicated to no purpose.  For example, now you have to check when
> attaching a partition that it doesn't violate the rule; otherwise you
> end up with a table that can't be created directly (and thus can't
> survive dump-and-restore) but can be created indirectly by exploiting
> loopholes in the checks.  It's tempting to think that we can check
> simple cases - e.g. if the parent and the child are partitioning on
> the same exact column, the child's range should be contained within
> the parent's range - but more complicated cases are tricky.  Suppose
> the table is range-partitioned on (a, b) and range-subpartitioned on
> b.  It's not trivial to figure out whether the set of values that the
> user can insert into that partition is non-empty.  If we allow
> partitioning on expressions, then it quickly becomes altogether
> impossible to deduce anything useful - unless you can solve the
> halting problem.
>
> And, really, why do we care?  If the user creates a partitioning
> scheme that permits no rows in some or all of the partitions, then
> they will have an empty table that can be correctly dumped and
> restored but which cannot be used for anything useful unless it is
> modified first.  They probably don't want that, but it's not any more
> broken than a table inheritance setup with mutually exclusive CHECK
> constraints, or for that matter a plain table with mutually exclusive
> CHECK constraints - and we don't try to prohibit those things.  This
> patch is supposed to be implementing partitioning, not artificial
> intelligence.

Agree with your arguments.  Certainly, I overlooked some use cases
that my proposed solution would outright prevent.  I withdraw it.

Thanks,
Amit

On Mon, Aug 15, 2016 at 7:21 AM, Amit Langote
<Langote_Amit_f8@lab.ntt.co.jp> wrote:
>> +    if (partexprs)
>> +        recordDependencyOnSingleRelExpr(&myself,
>> +                                        (Node *) partexprs,
>> +                                        RelationGetRelid(rel),
>> +                                        DEPENDENCY_NORMAL,
>> +                                        DEPENDENCY_IGNORE);
>>
>> I don't think introducing a new DEPENDENCY_IGNORE type is a good idea
>> here.  Instead, you could just add an additional Boolean argument to
>> recordDependencyOnSingleRelExpr.  That seems less likely to create
>> bugs in unrelated portions of the code.
>
> I did consider a Boolean argument instead of a new DependencyType value,
> however it felt a bit strange to pass a valid value for the fifth argument
> (self_behavior) and then ask using a separate parameter that it (a
> self-dependency) is to be ignored.  By the way, no pg_depend entry is
> created on such a call, so the effect of the new type's usage seems
> localized to me. Thoughts?

I think that's not a very plausible argument.  If you add a fifth
argument to that function, then only that function needs to know about
the possibility of ignoring self-dependencies.  If you add a new
dependency type, then everything that knows about DependencyType needs
to know about them.  That's got to be a much larger surface area for
bugs.  Also, if you look around a bit, I believe you will find other
examples of cases where one argument is used only for certain values
of some other argument.  That's not a novel design pattern.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Thu, Sep 15, 2016 at 4:53 AM, Amit Langote
<Langote_Amit_f8@lab.ntt.co.jp> wrote:
> [ new patches ]

Re-reviewing 0001.

+      <entry><structfield>partexprs</structfield></entry>
+      <entry><type>pg_node_tree</type></entry>

This documentation doesn't match pg_partition_table.h, which has
partexprsrc and partexprbin.  I don't understand why it's a good idea
to have both, and there seem to be no comments or documentation
supporting that choice anywhere.

+      The optional <literal>PARTITION BY</> clause specifies a method of
+      partitioning the table and the corresponding partition key.  Table
+      thus created is called <firstterm>partitioned</firstterm> table.  Key
+      consists of an ordered list of column names and/or expressions when
+      using the <literal>RANGE</> method, whereas only a single column or
+      expression can be specified when using the <literal>LIST</> method.
+      The type of a key column or an expression must have an associated
+      btree operator class or one must be specified along with the column
+      or the expression.

Both of the sentences in this paragraph that do not begin with "the"
need to begin with "the".  (In my experience, it's generally a feature
of English as spoken in India that connecting words like "the" and "a"
are sometimes left out where non-Indian speakers of English would
include them, so it would be good to watch out for this issue in
general.)

Also, I think this should be rephrased a bit to be more clear about
how the partitioning key works, like this: The optional
<literal>PARTITION BY</literal> clause specifies a method of
partitioning the table.  The table thus created is called a
<firstterm>partitioned</firstterm> table.  The parenthesized list of
expressions forms the <firsttem>partitioning key</firstterm> for the
table.  When using range partitioning, the partioning key can include
multiple columns or expressions, but for list partitioning, the
partitioning key must consist of a single column or expression.  If no
btree operator class is specified when creating a partitioned table,
the default btree operator class for the datatype will be used.  If
there is none, an error will be reported.

+        case RELKIND_PARTITIONED_TABLE:            options = heap_reloptions(classForm->relkind, datum, false);

Why?  None of the reloptions that pertain to heap seem relevant to a
relkind without storage.

But, ah, do partitioned tables have storage?  I mean, do we end up
with an empty file, or no relfilenode at all?  Can I CLUSTER, VACUUM,
etc. a partitioned table?  It would seem cleaner for the parent to
have no relfilenode at all, but I'm guessing we might need some more
changes for that to work out.

+    pg_collation.h pg_range.h pg_transform.h pg_partitioned_table.h\

Whitespace.  Also, here and elsewhere, how about using alphabetical
order, or anyway preserving it insofar as the existing list is
alphabetized?

+    /* Remove NO INHERIT flag if rel is a partitioned table */
+    if (is_no_inherit &&
+        rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
+        ereport(ERROR,
+                (errcode(ERRCODE_INVALID_TABLE_DEFINITION),
+                 errmsg("cannot add NO INHERIT constraint to
partitioned table \"%s\"",
+                         RelationGetRelationName(rel))));

The code and the comment disagree.  I think the code is right and the
comment should be adjusted to say something like /* Partitioned tables
do not have storage, so a NO INHERIT constraint makes no sense. */

+ * IDENTIFICATION
+ *        src/backend/utils/misc/partition.c

Wrong.

+} KeyTypeCollInfo;

I don't like this type name very much.  Can we get "Part" in there someplace?

It doesn't seem to be very well-designed, either.  The number of
entries in each array is determined by the partnatts flag in
PartitionKeyData, which has also got various other arrays whose
lengths are determined by partnatts.  Why do we have some arrays in
one structure and some arrays in another structure?  Would it hurt
anything to merge everything into one structure?  Or could
PartitionKeyData include a field of type KeyTypeCollInfo rather than
KeyTypeCollInfo *, saving one pointer reference every place we access
this data?

+    /* Allocate in the supposedly short-lived working context */

Why supposedly?

+    datum = fastgetattr(tuple, Anum_pg_partitioned_table_partattrs,
+                        RelationGetDescr(catalog),
+                        &isnull);

Isn't the point of putting the fixed-length fields first that we can
use GETSTRUCT() here?  And even for partattrs as the first
variable-length thing?

+        /*
+         * Run the expressions through eval_const_expressions. This is
+         * not just an optimization, but is necessary, because eventually
+         * the planner will be comparing them to similarly-processed qual
+         * clauses, and may fail to detect valid matches without this.
+         * We don't bother with canonicalize_qual, however.
+         */

I'm a bit confused by this, because I would think this processing
ought to have been done before storing anything in the system
catalogs.  I don't see why it should be necessary to do it again after
pulling data back out of the system catalogs.

+            Value *str = lfirst(partexprsrc_item);
+            key->partcolnames[i] = pstrdup(str->val.str);

Should have a blank line in between.

+/*
+ * Partition key information inquiry functions
+ */
+int
+get_partition_key_strategy(PartitionKey key)
+{
+    return key->strategy;
+}
+
+int
+get_partition_key_natts(PartitionKey key)
+{
+    return key->partnatts;
+}
+
+List *
+get_partition_key_exprs(PartitionKey key)
+{
+    return key->partexprs;
+}
+
+/*
+ * Partition key information inquiry functions - one column
+ */
+int16
+get_partition_col_attnum(PartitionKey key, int col)
+{
+    return key->partattrs[col];
+}
+
+Oid
+get_partition_col_typid(PartitionKey key, int col)
+{
+    return key->tcinfo->typid[col];
+}
+
+int32
+get_partition_col_typmod(PartitionKey key, int col)
+{
+    return key->tcinfo->typmod[col];
+}

If we're going to add notation for this, I think we should use macros
(or static inline functions defined in the header file).  Doing it
this way adds more cycles for no benefit.

+    newkey->partattrs = (AttrNumber *)
+                            palloc0(newkey->partnatts * sizeof(AttrNumber));
+    memcpy(newkey->partattrs, fromkey->partattrs,
+                            newkey->partnatts * sizeof(AttrNumber));

It's wasteful to use palloc0 if you're immediately going to overwrite
every byte in the array.  Use regular palloc instead.

+     * Copy the partition key, opclass info into arrays (should we
+     * make the caller pass them like this to start with?)

Only if it happens to be convenient for the caller, which doesn't seem
to be the case here.

+    /* Only this can ever be NULL */
+    if (!partexprbinDatum)
+    {
+        nulls[Anum_pg_partitioned_table_partexprbin - 1] = true;
+        nulls[Anum_pg_partitioned_table_partexprsrc - 1] = true;
+    }

How can it be valid to have no partitioning expressions?

+    /* Tell world about the key */
+    CacheInvalidateRelcache(rel);

Is this really needed?  Isn't the caller going to do something similar
pretty soon?

+    heap_freetuple(tuple);

Probably useless - might as well let the context reset clean it up.

+    simple_heap_delete(rel, &tuple->t_self);
+
+    /* Update the indexes on pg_partitioned_table */
+    CatalogUpdateIndexes(rel, tuple);

You don't need CatalogUpdateIndexes() after a delete, only after an
insert or update.

+    if (classform->relkind != relkind &&
+                (relkind == RELKIND_RELATION &&
+                    classform->relkind != RELKIND_PARTITIONED_TABLE))

That's broken.  Note that all of the conditions are joined using &&,
so if any one of them fails then we won't throw an error.  In
particular, it's no longer possible to throw an error when relkind is
not RELKIND_RELATION.

+/* Checks if a Var node is for a given attnum */
+static bool
+find_attr_reference_walker(Node *node, find_attr_reference_context *context)
+{
+    if (node == NULL)
+        return false;
+
+    if (IsA(node, Var))
+    {
+        Var       *var = (Var *) node;
+        char   *varattname = get_attname(context->relid, var->varattno);
+
+        if (!strcmp(varattname, context->attname))
+            return true;
+    }
+
+    return expression_tree_walker(node, find_attr_reference_walker, context);
+}

Hrm.  The comment says we're matching on attnum, but the code says
we're matching on attname.  is_partition_attr() has the same confusion
between comments and code.  Maybe instead of this whole approach it
would be better to use pull_varattnos(), then get_attnum() to find the
attribute number for the one you want, then bms_is_member().

+static PartitionBy *
+transformPartitionBy(Relation rel, PartitionBy *partitionby)
+{
+    PartitionBy       *partby;
+    ParseState       *pstate;
+    RangeTblEntry  *rte;
+    ListCell       *l;
+
+    partby = (PartitionBy *) makeNode(PartitionBy);
+
+    partby->strategy = partitionby->strategy;
+    partby->location = partitionby->location;
+    partby->partParams = NIL;
+
+    /*
+     * Create a dummy ParseState and insert the target relation as its sole
+     * rangetable entry.  We need a ParseState for transformExpr.
+     */
+    pstate = make_parsestate(NULL);

Why isn't this logic being invoked from transformCreateStmt()?  Then
we could use the actual parseState for the query instead of a fake
one.

+            if (IsA(expr, CollateExpr))
+                ereport(ERROR,
+                        (errcode(ERRCODE_INVALID_OBJECT_DEFINITION),
+                         errmsg("cannot use COLLATE in partition key
expression")));

I assume there is a good reason for this seemingly-arbitrary
restriction, but there's no comment saying what it is.  One thing
that's odd is that this will only prohibit a CollateExpr at the top
level, not in some more-deeply nested position.  That seems
suspicious.

+                /*
+                 * User wrote "(column)" or "(column COLLATE something)".
+                 * Treat it like simple attribute anyway.
+                 */

Evidently, the user did not do that, because you just prohibited the
second one of those.

+                if (IsA(expr, Const))
+                    ereport(ERROR,
+                            (errcode(ERRCODE_INVALID_OBJECT_DEFINITION),
+                             errmsg("cannot use a constant expression
as partition key")));
+
+                if (contain_mutable_functions(expr))
+                    ereport(ERROR,
+                            (errcode(ERRCODE_INVALID_OBJECT_DEFINITION),
+                             errmsg("functions in partition key
expression must be marked IMMUTABLE")));

Do these checks parallel what we do for CHECK constraints?  It might
be good to apply about the same level of rigor in both cases.

+                exprsrc = deparse_expression(expr,
+                            deparse_context_for(RelationGetRelationName(rel),
+                                                RelationGetRelid(rel)),
+                                       false, false);

Why bother?  The output of this doesn't seem like a useful thing to
store.  The fact that we've done similar things elsewhere doesn't make
it a good idea.  I think we did it in other cases because we used to
be dumber than we are now.

+                        (errcode(ERRCODE_UNDEFINED_OBJECT),
+                         errmsg("data type %s has no default btree
operator class",
+                                format_type_be(atttype)),
+                         errhint("You must specify an existing btree
operator class or define one for the type.")));

The hint is not really accurate, because the type may well have a
btree operator class.  Just not a default one.

+                            char    relkind = ((CreateStmt *)
stmt)->partby != NULL
+                                                    ? RELKIND_PARTITIONED_TABLE
+                                                    : RELKIND_RELATION;

Let's push this down into DefineRelation().  i.e. if (stmt->partby !=
NULL) { if (relkind != RELKIND_RELATION) ereport(...); relkind =
RELKIND_PARTITION_TABLE; }

+        RelationBuildPartitionKey(relation);

I wonder if RelationBuildPartitionKey should really be in relcache.c.
What do we do in similar cases?

+} PartitionBy;

Maybe PartitionSpec?

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Hi Amit,
Following sequence of DDLs gets an error
--
-- multi-leveled partitions
--
CREATE TABLE prt1_l (a int, b int, c varchar) PARTITION BY RANGE(a);
CREATE TABLE prt1_l_p1 PARTITION OF prt1_l FOR VALUES START (0) END
(250) PARTITION BY RANGE (b);
CREATE TABLE prt1_l_p1_p1 PARTITION OF prt1_l_p1 FOR VALUES START (0) END (100);
CREATE TABLE prt1_l_p1_p2 PARTITION OF prt1_l_p1 FOR VALUES START
(100) END (250);
CREATE TABLE prt1_l_p2 PARTITION OF prt1_l FOR VALUES START (250) END
(500) PARTITION BY RANGE (c);
CREATE TABLE prt1_l_p2_p1 PARTITION OF prt1_l_p2 FOR VALUES START
('0250') END ('0400');
CREATE TABLE prt1_l_p2_p2 PARTITION OF prt1_l_p2 FOR VALUES START
('0400') END ('0500');
CREATE TABLE prt1_l_p3 PARTITION OF prt1_l FOR VALUES START (500) END
(600) PARTITION BY RANGE ((b + a));
ERROR:  cannot use column or expression from ancestor partition key

The last statement is trying create subpartitions by range (b + a),
which contains a partition key from ancestor partition key but is not
exactly same as that. In fact it contains some extra columns other
than the ancestor partition key columns. Why do we want to prohibit
such cases?

On Thu, Sep 15, 2016 at 2:23 PM, Amit Langote
<Langote_Amit_f8@lab.ntt.co.jp> wrote:
>
> Hi
>
> On 2016/09/09 17:55, Amit Langote wrote:
>> On 2016/09/06 22:04, Amit Langote wrote:
>>> Will fix.
>>
>> Here is an updated set of patches.
>
> An email from Rajkumar somehow managed to break out of this thread.
> Quoting his message below so that I don't end up replying with patches on
> two different threads.
>
> On 2016/09/14 16:58, Rajkumar Raghuwanshi wrote:
>> I have Continued with testing declarative partitioning with the latest
>> patch. Got some more observation, given below
>
> Thanks a lot for testing.
>
>> -- Observation 1 : Getting overlap error with START with EXCLUSIVE in range
>> partition.
>>
>> create table test_range_bound ( a int) partition by range(a);
>> --creating a partition to contain records {1,2,3,4}, by default 1 is
>> inclusive and 5 is exclusive
>> create table test_range_bound_p1 partition of test_range_bound for values
>> start (1) end (5);
>> --now trying to create a partition by explicitly mentioning start is
>> exclusive to contain records {5,6,7}, here trying to create with START with
>> 4 as exclusive so range should be 5 to 8, but getting partition overlap
>> error.
>> create table test_range_bound_p2 partition of test_range_bound for values
>> start (4) EXCLUSIVE end (8);
>> ERROR:  partition "test_range_bound_p2" would overlap partition
>> "test_range_bound_p1"
>
> Wow, this is bad.  What is needed in this case is "canonicalization" of
> the range partition bounds specified in the command.  Range types do this
> and hence an equivalent test done with range type values would disagree
> with the result given by the patch for range partition bounds.
>
> select '[1,5)'::int4range && '(4,8]'::int4range as cmp;
>  cmp
> -----
>  f
> (1 row)
>
> In this case, the second range is converted into its equivalent canonical
> form viz. '[5, 9)'.  Then comparison of bounds 5) and [5 can tell that the
> ranges do not overlap after all.  Range type operators can do this because
> their code can rely on the availability of a canonicalization function for
> a given range type.  From the range types documentation:
>
> """
> A discrete range type should have a canonicalization function that is
> aware of the desired step size for the element type. The canonicalization
> function is charged with converting equivalent values of the range type to
> have identical representations, in particular consistently inclusive or
> exclusive bounds. If a canonicalization function is not specified, then
> ranges with different formatting will always be treated as unequal, even
> though they might represent the same set of values in reality.
> """
>
> to extend the last sentence:
>
> "... or consider two ranges overlapping when in reality they are not
> (maybe they are really just adjacent)."
>
> Within the code handling range partition bound, no such canonicalization
> happens, so comparison 5) and (4 ends up concluding that upper1 > lower2,
> hence ranges overlap.
>
> To mitigate this, how about we restrict range partition key to contain
> columns of only those types for which we know we can safely canonicalize a
> range bound (ie, discrete range types)?  I don't think we can use, say,
> existing int4range_canonical but will have to write a version of it for
> partitioning usage (range bounds of partitions are different from what
> int4range_canonical is ready to handle).  This approach will be very
> limiting as then range partitions will be limited to columns of int,
> bigint and date type only.
>
> One more option is we let the user specify the canonicalize function next
> to the column name when defining the partition key.  If not specified, we
> hard-code one for the types for which we will be implementing a
> canonicalize function (ie, above mentioned types).  In other cases, we
> just don't have one and hence if an unexpected result occurs when creating
> a new partition, it's up to the user to realize what happened.  Of course,
> we will be mentioning in the documentation why a canonicalize function is
> necessary and how to write one.  Note that this canonicalize function
> comes into play only when defining new partitions, it has no role beyond
> that point.
>
>> -- Observation 2 : able to create sub-partition out of the range set for
>> main table, causing not able to insert data satisfying any of the partition.
>>
>> create table test_subpart (c1 int) partition by range (c1);
>> create table test_subpart_p1 partition of test_subpart for values start (1)
>> end (100) inclusive partition by range (c1);
>> create table test_subpart_p1_sub1 partition of test_subpart_p1 for values
>> start (101) end (200);
>>
>> \d+ test_subpart
>>                      Table "public.test_subpart"
>>  Column |  Type   | Modifiers | Storage | Stats target | Description
>> --------+---------+-----------+---------+--------------+-------------
>>  c1     | integer |           | plain   |              |
>> Partition Key: RANGE (c1)
>> Partitions: test_subpart_p1 FOR VALUES START (1) END (100) INCLUSIVE
>>
>> \d+ test_subpart_p1
>>                    Table "public.test_subpart_p1"
>>  Column |  Type   | Modifiers | Storage | Stats target | Description
>> --------+---------+-----------+---------+--------------+-------------
>>  c1     | integer |           | plain   |              |
>> Partition Of: test_subpart FOR VALUES START (1) END (100) INCLUSIVE
>> Partition Key: RANGE (c1)
>> Partitions: test_subpart_p1_sub1 FOR VALUES START (101) END (200)
>>
>> insert into test_subpart values (50);
>> ERROR:  no partition of relation "test_subpart_p1" found for row
>> DETAIL:  Failing row contains (50).
>> insert into test_subpart values (150);
>> ERROR:  no partition of relation "test_subpart" found for row
>> DETAIL:  Failing row contains (150).
>
> It seems that DDL should prevent the same column being used in partition
> key of lower level partitions.  I don't know how much sense it would make,
> but being able to use the same column as partition key of lower level
> partitions may be a feature useful to some users if they know what they
> are doing.  But this last part doesn't sound like a good thing.  I
> modified the patch such that lower level partitions cannot use columns
> used by ancestor tables.
>
>> -- Observation 3 : Getting cache lookup failed, when selecting list
>> partition table containing array.
>>
>> CREATE TABLE test_array ( i int,j int[],k text[]) PARTITION BY LIST (j);
>> CREATE TABLE test_array_p1 PARTITION OF test_array FOR VALUES IN ('{1}');
>> CREATE TABLE test_array_p2 PARTITION OF test_array FOR VALUES IN ('{2,2}');
>>
>> INSERT INTO test_array (i,j[1],k[1]) VALUES (1,1,1);
>> INSERT INTO test_array (i,j[1],j[2],k[1]) VALUES (2,2,2,2);
>>
>> postgres=# SELECT tableoid::regclass,* FROM test_array_p1;
>>    tableoid    | i |  j  |  k
>> ---------------+---+-----+-----
>>  test_array_p1 | 1 | {1} | {1}
>> (1 row)
>>
>> postgres=# SELECT tableoid::regclass,* FROM test_array_p2;
>>    tableoid    | i |   j   |  k
>> ---------------+---+-------+-----
>>  test_array_p2 | 2 | {2,2} | {2}
>> (1 row)
>>
>> postgres=# SELECT tableoid::regclass,* FROM test_array;
>> ERROR:  cache lookup failed for type 0
>
> That's a bug.  Fixed in the attached patch.
>
> PS: I'm going to have limited Internet access during this weekend and over
> the next week, so responses could be slow.  Sorry about that.
>
> Thanks,
> Amit



-- 
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company

For list partitions, the ListInfo stores the index maps for values
i.e. the index of the partition to which the value belongs. Those
indexes are same as the indexes in partition OIDs array and come from
the catalogs. In case a user creates two partitioned tables with
exactly same lists for partitions but specifies them in a different
order, the OIDs are stored in the order specified. This means that
index array for these tables come out different. equal_list_info()
works around that by creating an array of mappings and checks whether
that mapping is consistent for all values. This means we will create
the mapping as many times as equal_list_info() is called, which is
expected to be more than the number of time
RelationBuildPartitionDescriptor() is called. Instead, if we
"canonicalise" the indexes so that they come out exactly same for
similarly partitioned tables, we build the mapping only once and
arrange OIDs accordingly.

Here's patch to do that. I have ran make check with this and it didn't
show any failure. Please consider this to be included in your next set
of patches.

That helps partition-wise join as well. For partition-wise join (and
further optimizations for partitioned tables), we create a list of
canonical partition schemes. In this list two similarly partitioned
tables share partition scheme pointer. A join between relations with
same partition scheme pointer can be joined partition-wise. It's
important that the indexes in partition scheme match to the OIDs array
to find matching RelOptInfos for partition-wise join.

On Thu, Sep 22, 2016 at 11:12 AM, Ashutosh Bapat
<ashutosh.bapat@enterprisedb.com> wrote:
> Hi Amit,
> Following sequence of DDLs gets an error
> --
> -- multi-leveled partitions
> --
> CREATE TABLE prt1_l (a int, b int, c varchar) PARTITION BY RANGE(a);
> CREATE TABLE prt1_l_p1 PARTITION OF prt1_l FOR VALUES START (0) END
> (250) PARTITION BY RANGE (b);
> CREATE TABLE prt1_l_p1_p1 PARTITION OF prt1_l_p1 FOR VALUES START (0) END (100);
> CREATE TABLE prt1_l_p1_p2 PARTITION OF prt1_l_p1 FOR VALUES START
> (100) END (250);
> CREATE TABLE prt1_l_p2 PARTITION OF prt1_l FOR VALUES START (250) END
> (500) PARTITION BY RANGE (c);
> CREATE TABLE prt1_l_p2_p1 PARTITION OF prt1_l_p2 FOR VALUES START
> ('0250') END ('0400');
> CREATE TABLE prt1_l_p2_p2 PARTITION OF prt1_l_p2 FOR VALUES START
> ('0400') END ('0500');
> CREATE TABLE prt1_l_p3 PARTITION OF prt1_l FOR VALUES START (500) END
> (600) PARTITION BY RANGE ((b + a));
> ERROR:  cannot use column or expression from ancestor partition key
>
> The last statement is trying create subpartitions by range (b + a),
> which contains a partition key from ancestor partition key but is not
> exactly same as that. In fact it contains some extra columns other
> than the ancestor partition key columns. Why do we want to prohibit
> such cases?
>
> On Thu, Sep 15, 2016 at 2:23 PM, Amit Langote
> <Langote_Amit_f8@lab.ntt.co.jp> wrote:
>>
>> Hi
>>
>> On 2016/09/09 17:55, Amit Langote wrote:
>>> On 2016/09/06 22:04, Amit Langote wrote:
>>>> Will fix.
>>>
>>> Here is an updated set of patches.
>>
>> An email from Rajkumar somehow managed to break out of this thread.
>> Quoting his message below so that I don't end up replying with patches on
>> two different threads.
>>
>> On 2016/09/14 16:58, Rajkumar Raghuwanshi wrote:
>>> I have Continued with testing declarative partitioning with the latest
>>> patch. Got some more observation, given below
>>
>> Thanks a lot for testing.
>>
>>> -- Observation 1 : Getting overlap error with START with EXCLUSIVE in range
>>> partition.
>>>
>>> create table test_range_bound ( a int) partition by range(a);
>>> --creating a partition to contain records {1,2,3,4}, by default 1 is
>>> inclusive and 5 is exclusive
>>> create table test_range_bound_p1 partition of test_range_bound for values
>>> start (1) end (5);
>>> --now trying to create a partition by explicitly mentioning start is
>>> exclusive to contain records {5,6,7}, here trying to create with START with
>>> 4 as exclusive so range should be 5 to 8, but getting partition overlap
>>> error.
>>> create table test_range_bound_p2 partition of test_range_bound for values
>>> start (4) EXCLUSIVE end (8);
>>> ERROR:  partition "test_range_bound_p2" would overlap partition
>>> "test_range_bound_p1"
>>
>> Wow, this is bad.  What is needed in this case is "canonicalization" of
>> the range partition bounds specified in the command.  Range types do this
>> and hence an equivalent test done with range type values would disagree
>> with the result given by the patch for range partition bounds.
>>
>> select '[1,5)'::int4range && '(4,8]'::int4range as cmp;
>>  cmp
>> -----
>>  f
>> (1 row)
>>
>> In this case, the second range is converted into its equivalent canonical
>> form viz. '[5, 9)'.  Then comparison of bounds 5) and [5 can tell that the
>> ranges do not overlap after all.  Range type operators can do this because
>> their code can rely on the availability of a canonicalization function for
>> a given range type.  From the range types documentation:
>>
>> """
>> A discrete range type should have a canonicalization function that is
>> aware of the desired step size for the element type. The canonicalization
>> function is charged with converting equivalent values of the range type to
>> have identical representations, in particular consistently inclusive or
>> exclusive bounds. If a canonicalization function is not specified, then
>> ranges with different formatting will always be treated as unequal, even
>> though they might represent the same set of values in reality.
>> """
>>
>> to extend the last sentence:
>>
>> "... or consider two ranges overlapping when in reality they are not
>> (maybe they are really just adjacent)."
>>
>> Within the code handling range partition bound, no such canonicalization
>> happens, so comparison 5) and (4 ends up concluding that upper1 > lower2,
>> hence ranges overlap.
>>
>> To mitigate this, how about we restrict range partition key to contain
>> columns of only those types for which we know we can safely canonicalize a
>> range bound (ie, discrete range types)?  I don't think we can use, say,
>> existing int4range_canonical but will have to write a version of it for
>> partitioning usage (range bounds of partitions are different from what
>> int4range_canonical is ready to handle).  This approach will be very
>> limiting as then range partitions will be limited to columns of int,
>> bigint and date type only.
>>
>> One more option is we let the user specify the canonicalize function next
>> to the column name when defining the partition key.  If not specified, we
>> hard-code one for the types for which we will be implementing a
>> canonicalize function (ie, above mentioned types).  In other cases, we
>> just don't have one and hence if an unexpected result occurs when creating
>> a new partition, it's up to the user to realize what happened.  Of course,
>> we will be mentioning in the documentation why a canonicalize function is
>> necessary and how to write one.  Note that this canonicalize function
>> comes into play only when defining new partitions, it has no role beyond
>> that point.
>>
>>> -- Observation 2 : able to create sub-partition out of the range set for
>>> main table, causing not able to insert data satisfying any of the partition.
>>>
>>> create table test_subpart (c1 int) partition by range (c1);
>>> create table test_subpart_p1 partition of test_subpart for values start (1)
>>> end (100) inclusive partition by range (c1);
>>> create table test_subpart_p1_sub1 partition of test_subpart_p1 for values
>>> start (101) end (200);
>>>
>>> \d+ test_subpart
>>>                      Table "public.test_subpart"
>>>  Column |  Type   | Modifiers | Storage | Stats target | Description
>>> --------+---------+-----------+---------+--------------+-------------
>>>  c1     | integer |           | plain   |              |
>>> Partition Key: RANGE (c1)
>>> Partitions: test_subpart_p1 FOR VALUES START (1) END (100) INCLUSIVE
>>>
>>> \d+ test_subpart_p1
>>>                    Table "public.test_subpart_p1"
>>>  Column |  Type   | Modifiers | Storage | Stats target | Description
>>> --------+---------+-----------+---------+--------------+-------------
>>>  c1     | integer |           | plain   |              |
>>> Partition Of: test_subpart FOR VALUES START (1) END (100) INCLUSIVE
>>> Partition Key: RANGE (c1)
>>> Partitions: test_subpart_p1_sub1 FOR VALUES START (101) END (200)
>>>
>>> insert into test_subpart values (50);
>>> ERROR:  no partition of relation "test_subpart_p1" found for row
>>> DETAIL:  Failing row contains (50).
>>> insert into test_subpart values (150);
>>> ERROR:  no partition of relation "test_subpart" found for row
>>> DETAIL:  Failing row contains (150).
>>
>> It seems that DDL should prevent the same column being used in partition
>> key of lower level partitions.  I don't know how much sense it would make,
>> but being able to use the same column as partition key of lower level
>> partitions may be a feature useful to some users if they know what they
>> are doing.  But this last part doesn't sound like a good thing.  I
>> modified the patch such that lower level partitions cannot use columns
>> used by ancestor tables.
>>
>>> -- Observation 3 : Getting cache lookup failed, when selecting list
>>> partition table containing array.
>>>
>>> CREATE TABLE test_array ( i int,j int[],k text[]) PARTITION BY LIST (j);
>>> CREATE TABLE test_array_p1 PARTITION OF test_array FOR VALUES IN ('{1}');
>>> CREATE TABLE test_array_p2 PARTITION OF test_array FOR VALUES IN ('{2,2}');
>>>
>>> INSERT INTO test_array (i,j[1],k[1]) VALUES (1,1,1);
>>> INSERT INTO test_array (i,j[1],j[2],k[1]) VALUES (2,2,2,2);
>>>
>>> postgres=# SELECT tableoid::regclass,* FROM test_array_p1;
>>>    tableoid    | i |  j  |  k
>>> ---------------+---+-----+-----
>>>  test_array_p1 | 1 | {1} | {1}
>>> (1 row)
>>>
>>> postgres=# SELECT tableoid::regclass,* FROM test_array_p2;
>>>    tableoid    | i |   j   |  k
>>> ---------------+---+-------+-----
>>>  test_array_p2 | 2 | {2,2} | {2}
>>> (1 row)
>>>
>>> postgres=# SELECT tableoid::regclass,* FROM test_array;
>>> ERROR:  cache lookup failed for type 0
>>
>> That's a bug.  Fixed in the attached patch.
>>
>> PS: I'm going to have limited Internet access during this weekend and over
>> the next week, so responses could be slow.  Sorry about that.
>>
>> Thanks,
>> Amit
>
>
>
> --
> Best Wishes,
> Ashutosh Bapat
> EnterpriseDB Corporation
> The Postgres Database Company



--
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company

On Thu, Sep 22, 2016 at 1:02 PM, Ashutosh Bapat
<ashutosh.bapat@enterprisedb.com> wrote:
> For list partitions, the ListInfo stores the index maps for values
> i.e. the index of the partition to which the value belongs. Those
> indexes are same as the indexes in partition OIDs array and come from
> the catalogs. In case a user creates two partitioned tables with
> exactly same lists for partitions but specifies them in a different
> order, the OIDs are stored in the order specified. This means that
> index array for these tables come out different. equal_list_info()
> works around that by creating an array of mappings and checks whether
> that mapping is consistent for all values. This means we will create
> the mapping as many times as equal_list_info() is called, which is
> expected to be more than the number of time
> RelationBuildPartitionDescriptor() is called. Instead, if we
> "canonicalise" the indexes so that they come out exactly same for
> similarly partitioned tables, we build the mapping only once and
> arrange OIDs accordingly.
>
> Here's patch to do that. I have ran make check with this and it didn't
> show any failure. Please consider this to be included in your next set
> of patches.

The patch has an if condition as statement by itself
+    if (l1->null_index != l2->null_index);        return false;

There shouldn't be ';' at the end. It looks like in the tests you have
added the function always bails out before it reaches this statement.

-- 
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company

Hi Ashutosh,

On 2016/09/22 14:42, Ashutosh Bapat wrote:
> Hi Amit,
> Following sequence of DDLs gets an error
> --
> -- multi-leveled partitions
> --
> CREATE TABLE prt1_l (a int, b int, c varchar) PARTITION BY RANGE(a);
> CREATE TABLE prt1_l_p1 PARTITION OF prt1_l FOR VALUES START (0) END
> (250) PARTITION BY RANGE (b);
> CREATE TABLE prt1_l_p1_p1 PARTITION OF prt1_l_p1 FOR VALUES START (0) END (100);
> CREATE TABLE prt1_l_p1_p2 PARTITION OF prt1_l_p1 FOR VALUES START
> (100) END (250);
> CREATE TABLE prt1_l_p2 PARTITION OF prt1_l FOR VALUES START (250) END
> (500) PARTITION BY RANGE (c);
> CREATE TABLE prt1_l_p2_p1 PARTITION OF prt1_l_p2 FOR VALUES START
> ('0250') END ('0400');
> CREATE TABLE prt1_l_p2_p2 PARTITION OF prt1_l_p2 FOR VALUES START
> ('0400') END ('0500');
> CREATE TABLE prt1_l_p3 PARTITION OF prt1_l FOR VALUES START (500) END
> (600) PARTITION BY RANGE ((b + a));
> ERROR:  cannot use column or expression from ancestor partition key
> 
> The last statement is trying create subpartitions by range (b + a),
> which contains a partition key from ancestor partition key but is not
> exactly same as that. In fact it contains some extra columns other
> than the ancestor partition key columns. Why do we want to prohibit
> such cases?

Per discussion [1], I am going to remove this ill-considered restriction.

Thanks,
Amit

[1]
https://www.postgresql.org/message-id/CA%2BHiwqEXAU_m%2BV%3Db-VGmsDNjoqc-Z_9KQdyPuOGbiQGzNObmVg%40mail.gmail.com

On 2016/09/22 19:10, Ashutosh Bapat wrote:
> On Thu, Sep 22, 2016 at 1:02 PM, Ashutosh Bapat
> <ashutosh.bapat@enterprisedb.com> wrote:
>> For list partitions, the ListInfo stores the index maps for values
>> i.e. the index of the partition to which the value belongs. Those
>> indexes are same as the indexes in partition OIDs array and come from
>> the catalogs. In case a user creates two partitioned tables with
>> exactly same lists for partitions but specifies them in a different
>> order, the OIDs are stored in the order specified. This means that
>> index array for these tables come out different. equal_list_info()
>> works around that by creating an array of mappings and checks whether
>> that mapping is consistent for all values. This means we will create
>> the mapping as many times as equal_list_info() is called, which is
>> expected to be more than the number of time
>> RelationBuildPartitionDescriptor() is called. Instead, if we
>> "canonicalise" the indexes so that they come out exactly same for
>> similarly partitioned tables, we build the mapping only once and
>> arrange OIDs accordingly.
>>
>> Here's patch to do that. I have ran make check with this and it didn't
>> show any failure. Please consider this to be included in your next set
>> of patches.

Thanks.  It seems like this will save quite a few cycles for future users
of equal_list_info().  I will incorporate it into may patch.

With this patch, the mapping is created *only once* during
RelationBuildPartitionDesc() to assign canonical indexes to individual
list values.  The partition OID array will also be rearranged such that
using the new (canonical) index instead of the old
catalog-scan-order-based index will retrieve the correct partition for
that value.

By the way, I fixed one thinko in your patch as follows:

-        result->oids[i] = oids[mapping[i]];
+        result->oids[mapping[i]] = oids[i];

That is, copy an OID at a given position in the original array to a
*mapped position* in the result array (instead of the other way around).

> The patch has an if condition as statement by itself
> +    if (l1->null_index != l2->null_index);
>          return false;
> 
> There shouldn't be ';' at the end. It looks like in the tests you have
> added the function always bails out before it reaches this statement.

There is no user of equal_list_info() such that the above bug would have
caused a regression test failure.  Maybe a segfault crash (due to dangling
pointer into partition descriptor's listinfo after the above function
would unintentionally return false to equalPartitionDescs() which is in
turn called by RelationClearRelation()).

Thanks,
Amit

>
> With this patch, the mapping is created *only once* during
> RelationBuildPartitionDesc() to assign canonical indexes to individual
> list values.  The partition OID array will also be rearranged such that
> using the new (canonical) index instead of the old
> catalog-scan-order-based index will retrieve the correct partition for
> that value.
>
> By the way, I fixed one thinko in your patch as follows:
>
> -        result->oids[i] = oids[mapping[i]];
> +        result->oids[mapping[i]] = oids[i];

While I can not spot any problem with this logic, when I make that
change and run partition_join testcase in my patch, it fails because
wrong partitions are matched for partition-wise join of list
partitions. In that patch, RelOptInfo of partitions are saved in
RelOptInfo of the parent by matching their OIDs. They are saved in the
same order as corresponding OIDs. Partition-wise join simply joins the
RelOptInfos at the same positions from both the parent RelOptInfos. I
can not spot an error in this logic too.

-- 
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company

Sorry about the delay in replying.

On 2016/09/15 21:58, Ashutosh Bapat wrote:
> Hi Amit,
>
> It looks like there is some problem while creating paramterized paths
> for multi-level partitioned tables. Here's a longish testcase
>
> [ ... ]
>
> Please check if you are able to reproduce these errors in your
> repository. I made sure that I cleaned up all partition-wise join code
> before testing this, but ... .

Thanks for the test case.  I can reproduce the same.

> I tried to debug the problem somewhat. In set_append_rel_pathlist(),
> it finds that at least one child has a parameterized path as the
> cheapest path, so it doesn't create an unparameterized path for append
> rel. At the same time there is a parameterization common to all the
> children, so it doesn't create any path. There seem to be two problems
> here
> 1. The children from second level onwards may not be getting
> parameterized for lateral references. That seems unlikely but
> possible.
> 2. Reparameterization should have corrected this, but
> reparameterize_path() does not support AppendPaths.

Hmm, 0005-Refactor-optimizer-s-inheritance-set-expansion-code-5.patch is
certainly to be blamed here; if I revert the patch, the problem goes away.

Based on 2 above, I attempted to add logic for AppendPath in
reparameterize_path() as in the attached.  It fixes the reported problem
and does not break any regression tests. If it's sane to do things this
way, I will incorporate the attached into patch 0005 mentioned above.
Thoughts?

Thanks,
Amit

>>
>> Please check if you are able to reproduce these errors in your
>> repository. I made sure that I cleaned up all partition-wise join code
>> before testing this, but ... .
>
> Thanks for the test case.  I can reproduce the same.
>
>> I tried to debug the problem somewhat. In set_append_rel_pathlist(),
>> it finds that at least one child has a parameterized path as the
>> cheapest path, so it doesn't create an unparameterized path for append
>> rel. At the same time there is a parameterization common to all the
>> children, so it doesn't create any path. There seem to be two problems
>> here
>> 1. The children from second level onwards may not be getting
>> parameterized for lateral references. That seems unlikely but
>> possible.

Did you check this? We may be missing on creating index scan paths
with parameterization. If we fix this, we don't need to
re-parameterize Append.

>> 2. Reparameterization should have corrected this, but
>> reparameterize_path() does not support AppendPaths.
>
> Hmm, 0005-Refactor-optimizer-s-inheritance-set-expansion-code-5.patch is
> certainly to be blamed here; if I revert the patch, the problem goes away.
>
> Based on 2 above, I attempted to add logic for AppendPath in
> reparameterize_path() as in the attached.  It fixes the reported problem
> and does not break any regression tests. If it's sane to do things this
> way, I will incorporate the attached into patch 0005 mentioned above.
> Thoughts?


-- 
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company

On 2016/09/27 15:44, Ashutosh Bapat wrote:
>> By the way, I fixed one thinko in your patch as follows:
>>
>> -        result->oids[i] = oids[mapping[i]];
>> +        result->oids[mapping[i]] = oids[i];
> 
> While I can not spot any problem with this logic, when I make that
> change and run partition_join testcase in my patch, it fails because
> wrong partitions are matched for partition-wise join of list
> partitions. In that patch, RelOptInfo of partitions are saved in
> RelOptInfo of the parent by matching their OIDs. They are saved in the
> same order as corresponding OIDs. Partition-wise join simply joins the
> RelOptInfos at the same positions from both the parent RelOptInfos. I
> can not spot an error in this logic too.

OTOH, using the original logic makes tuple routing put tuples into the
wrong partitions.  When debugging why that was happening I discovered this
and hence the proposed change.

You mean that partition RelOptInfo's are placed using the canonical
ordering of OIDs instead of catalog-scan-driven order, right?  If that's
true, then there is no possibility of wrong pairing happening, even with
the new ordering of OIDs in the partition descriptor (ie, the ordering
that would be produced by my proposed method above).

Thanks,
Amit

On Tue, Sep 27, 2016 at 2:46 PM, Amit Langote
<Langote_Amit_f8@lab.ntt.co.jp> wrote:
> On 2016/09/27 15:44, Ashutosh Bapat wrote:
>>> By the way, I fixed one thinko in your patch as follows:
>>>
>>> -        result->oids[i] = oids[mapping[i]];
>>> +        result->oids[mapping[i]] = oids[i];
>>
>> While I can not spot any problem with this logic, when I make that
>> change and run partition_join testcase in my patch, it fails because
>> wrong partitions are matched for partition-wise join of list
>> partitions. In that patch, RelOptInfo of partitions are saved in
>> RelOptInfo of the parent by matching their OIDs. They are saved in the
>> same order as corresponding OIDs. Partition-wise join simply joins the
>> RelOptInfos at the same positions from both the parent RelOptInfos. I
>> can not spot an error in this logic too.
>
> OTOH, using the original logic makes tuple routing put tuples into the
> wrong partitions.  When debugging why that was happening I discovered this
> and hence the proposed change.
>
> You mean that partition RelOptInfo's are placed using the canonical
> ordering of OIDs instead of catalog-scan-driven order, right?  If that's
> true, then there is no possibility of wrong pairing happening, even with
> the new ordering of OIDs in the partition descriptor (ie, the ordering
> that would be produced by my proposed method above).

right! I don't know what's wrong, will debug my changes.


-- 
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company

On 2016/09/27 18:09, Ashutosh Bapat wrote:
>>> I tried to debug the problem somewhat. In set_append_rel_pathlist(),
>>> it finds that at least one child has a parameterized path as the
>>> cheapest path, so it doesn't create an unparameterized path for append
>>> rel. At the same time there is a parameterization common to all the
>>> children, so it doesn't create any path. There seem to be two problems
>>> here
>>> 1. The children from second level onwards may not be getting
>>> parameterized for lateral references. That seems unlikely but
>>> possible.
>
> Did you check this? We may be missing on creating index scan paths
> with parameterization. If we fix this, we don't need to
> re-parameterize Append.

You're right.  How about the attached patch that fixes the problem along
these lines?  The problem seems to be that multi-level inheritance sets
(partitioned tables) are not handled in create_lateral_join_info(), which
means that lateral_relids and lateral_referencers of the root relation are
not being propagated to the partitions below level 1.

I'm getting concerned about one thing though - for a given *regular*
inheritance set, the root->append_rel_list would be scanned only once; But
for a *partitioned table* inheritance set, it would be scanned for every
partitioned table in the set (ie, the root table and internal partitions).

Thanks,
Amit

Thanks a lot for the review and sorry about the delay in replying.
Combining responses to two emails.

On 2016/09/20 5:06, Robert Haas wrote:
> On Mon, Aug 15, 2016 at 7:21 AM, Amit Langote
> <Langote_Amit_f8@lab.ntt.co.jp> wrote:
>>> +    if (partexprs)
>>> +        recordDependencyOnSingleRelExpr(&myself,
>>> +                                        (Node *) partexprs,
>>> +                                        RelationGetRelid(rel),
>>> +                                        DEPENDENCY_NORMAL,
>>> +                                        DEPENDENCY_IGNORE);
>>>
>>> I don't think introducing a new DEPENDENCY_IGNORE type is a good idea
>>> here.  Instead, you could just add an additional Boolean argument to
>>> recordDependencyOnSingleRelExpr.  That seems less likely to create
>>> bugs in unrelated portions of the code.
>>
>> I did consider a Boolean argument instead of a new DependencyType value,
>> however it felt a bit strange to pass a valid value for the fifth argument
>> (self_behavior) and then ask using a separate parameter that it (a
>> self-dependency) is to be ignored.  By the way, no pg_depend entry is
>> created on such a call, so the effect of the new type's usage seems
>> localized to me. Thoughts?
>
> I think that's not a very plausible argument.  If you add a fifth
> argument to that function, then only that function needs to know about
> the possibility of ignoring self-dependencies.  If you add a new
> dependency type, then everything that knows about DependencyType needs
> to know about them.  That's got to be a much larger surface area for
> bugs.  Also, if you look around a bit, I believe you will find other
> examples of cases where one argument is used only for certain values
> of some other argument.  That's not a novel design pattern.

I removed DEPENDENCY_IGNORE.  Does the following look good or am I still
missing something?

@@ -83,7 +78,6 @@ typedef enum DependencyType
     DEPENDENCY_EXTENSION = 'e',
     DEPENDENCY_AUTO_EXTENSION = 'x',
     DEPENDENCY_PIN = 'p',
-    DEPENDENCY_IGNORE = 'g'

@@ -194,7 +188,8 @@ extern void recordDependencyOnExpr(const ObjectAddress
*depender,
 extern void recordDependencyOnSingleRelExpr(const ObjectAddress *depender,
                                 Node *expr, Oid relId,
                                 DependencyType behavior,
-                                DependencyType self_behavior);
+                                DependencyType self_behavior,
+                                bool ignore_self);

@@ -1450,9 +1449,10 @@ recordDependencyOnSingleRelExpr(const ObjectAddress
*depender,
         context.addrs->numrefs = outrefs;

         /* Record the self-dependencies */
-        recordMultipleDependencies(depender,
-                                   self_addrs->refs, self_addrs->numrefs,
-                                   self_behavior);
+        if (!ignore_self)
+            recordMultipleDependencies(depender,
+                                       self_addrs->refs, self_addrs->numrefs,
+                                       self_behavior);

@@ -138,7 +138,7 @@ StorePartitionKey(Relation rel,
                                         (Node *) partexprbin,
                                         RelationGetRelid(rel),
                                         DEPENDENCY_NORMAL,
-                                        DEPENDENCY_IGNORE);
+                                        DEPENDENCY_NORMAL, true);

> Re-reviewing 0001.
>
> +      <entry><structfield>partexprs</structfield></entry>
> +      <entry><type>pg_node_tree</type></entry>
>
> This documentation doesn't match pg_partition_table.h, which has
> partexprsrc and partexprbin.  I don't understand why it's a good idea
> to have both, and there seem to be no comments or documentation
> supporting that choice anywhere.

Removed partexprsrc, I realized the design whereby it was required to be
stored in the catalog was a dubious one after all.

> +      The optional <literal>PARTITION BY</> clause specifies a method of
> +      partitioning the table and the corresponding partition key.  Table
> +      thus created is called <firstterm>partitioned</firstterm> table.  Key
> +      consists of an ordered list of column names and/or expressions when
> +      using the <literal>RANGE</> method, whereas only a single column or
> +      expression can be specified when using the <literal>LIST</> method.
> +      The type of a key column or an expression must have an associated
> +      btree operator class or one must be specified along with the column
> +      or the expression.
>
> Both of the sentences in this paragraph that do not begin with "the"
> need to begin with "the".  (In my experience, it's generally a feature
> of English as spoken in India that connecting words like "the" and "a"
> are sometimes left out where non-Indian speakers of English would
> include them, so it would be good to watch out for this issue in
> general.)

Thanks for the tip, will try to be careful (rules about a/an/the's can be
too subtle for me sometimes, so any help is much appreciated).

> Also, I think this should be rephrased a bit to be more clear about
> how the partitioning key works, like this: The optional
> <literal>PARTITION BY</literal> clause specifies a method of
> partitioning the table.  The table thus created is called a
> <firstterm>partitioned</firstterm> table.  The parenthesized list of
> expressions forms the <firsttem>partitioning key</firstterm> for the
> table.  When using range partitioning, the partioning key can include
> multiple columns or expressions, but for list partitioning, the
> partitioning key must consist of a single column or expression.  If no
> btree operator class is specified when creating a partitioned table,
> the default btree operator class for the datatype will be used.  If
> there is none, an error will be reported.

Revised text along these lines.

> +        case RELKIND_PARTITIONED_TABLE:
>              options = heap_reloptions(classForm->relkind, datum, false);
>
> Why?  None of the reloptions that pertain to heap seem relevant to a
> relkind without storage.
>
> But, ah, do partitioned tables have storage?  I mean, do we end up
> with an empty file, or no relfilenode at all?  Can I CLUSTER, VACUUM,
> etc. a partitioned table?  It would seem cleaner for the parent to
> have no relfilenode at all, but I'm guessing we might need some more
> changes for that to work out.

Things were that way initially, that is, the parent relations had no
relfilenode.  I abandoned that project however.  The storage-less parent
thing seemed pretty daunting to me to handle right away.  For example,
optimizer and the executor code needed to be taught about the parent rel
appendrel member that used to be included as part of the result of
scanning the inheritance set but was no longer.

That said, there are still diffs in the patch resulting from simply
informing various sites in the code that there is a new relkind
RELKIND_PARTITIONED_TABLE, most of the sites giving it the same treatment
as RELKIND_RELATION.  Except maybe that the new relkind serves as a
convenient shorthand for specifying to various modules that the relation
is a partitioned table that is in some regards to be treated differently
from the ordinary tables.

> +    pg_collation.h pg_range.h pg_transform.h pg_partitioned_table.h\
>
> Whitespace.  Also, here and elsewhere, how about using alphabetical
> order, or anyway preserving it insofar as the existing list is
> alphabetized?

Fixed.

> +    /* Remove NO INHERIT flag if rel is a partitioned table */
> +    if (is_no_inherit &&
> +        rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
> +        ereport(ERROR,
> +                (errcode(ERRCODE_INVALID_TABLE_DEFINITION),
> +                 errmsg("cannot add NO INHERIT constraint to
> partitioned table \"%s\"",
> +                         RelationGetRelationName(rel))));
>
> The code and the comment disagree.  I think the code is right and the
> comment should be adjusted to say something like /* Partitioned tables
> do not have storage, so a NO INHERIT constraint makes no sense. */

Rewrote the comment.

>
> + * IDENTIFICATION
> + *        src/backend/utils/misc/partition.c
>
> Wrong.

This file is no longer part of the 0001 patch since I moved the file's
only function to relcache.c as mentioned below.  Fixed in the later patch
nonetheless.

>
> +} KeyTypeCollInfo;
>
> I don't like this type name very much.  Can we get "Part" in there someplace?
>
> It doesn't seem to be very well-designed, either.  The number of
> entries in each array is determined by the partnatts flag in
> PartitionKeyData, which has also got various other arrays whose
> lengths are determined by partnatts.  Why do we have some arrays in
> one structure and some arrays in another structure?  Would it hurt
> anything to merge everything into one structure?  Or could
> PartitionKeyData include a field of type KeyTypeCollInfo rather than
> KeyTypeCollInfo *, saving one pointer reference every place we access
> this data?

I followed your advice to just move the typ* arrays into the
PartitionKeyData struct.

> +    /* Allocate in the supposedly short-lived working context */
>
> Why supposedly?

When writing the code, I was thinking of the following header comment that
I copied from somewhere:

 * Note that the partition key data attached to a relcache entry must be
 * stored CacheMemoryContext to ensure it survives as long as the relcache
 * entry. But we should be running in a less long-lived working context.
 * To avoid leaking cache memory if this routine fails partway through,
 * we build in working memory and then copy the completed structure into
 * cache memory.

Removed the comment as it is not very illuminating anymore.

> +    datum = fastgetattr(tuple, Anum_pg_partitioned_table_partattrs,
> +                        RelationGetDescr(catalog),
> +                        &isnull);
>
> Isn't the point of putting the fixed-length fields first that we can
> use GETSTRUCT() here?  And even for partattrs as the first
> variable-length thing?

Right, fixed.

>
> +        /*
> +         * Run the expressions through eval_const_expressions. This is
> +         * not just an optimization, but is necessary, because eventually
> +         * the planner will be comparing them to similarly-processed qual
> +         * clauses, and may fail to detect valid matches without this.
> +         * We don't bother with canonicalize_qual, however.
> +         */
>
> I'm a bit confused by this, because I would think this processing
> ought to have been done before storing anything in the system
> catalogs.  I don't see why it should be necessary to do it again after
> pulling data back out of the system catalogs.

The pattern matches what's done for other expressions that optimizer deals
with, such as CHECK, index key, and index predicate expressions.

> +            Value *str = lfirst(partexprsrc_item);
> +            key->partcolnames[i] = pstrdup(str->val.str);
>
> Should have a blank line in between.

Fixed.

> +/*
> + * Partition key information inquiry functions
> + */
> +int
> +get_partition_key_strategy(PartitionKey key)
> +{
> +    return key->strategy;
> +}
> +
> [ ... ]
> +
> +int32
> +get_partition_col_typmod(PartitionKey key, int col)
> +{
> +    return key->tcinfo->typmod[col];
> +}
>
> If we're going to add notation for this, I think we should use macros
> (or static inline functions defined in the header file).  Doing it
> this way adds more cycles for no benefit.

OK, done using the static inline functions defined in the header file way.

> +    newkey->partattrs = (AttrNumber *)
> +                            palloc0(newkey->partnatts * sizeof(AttrNumber));
> +    memcpy(newkey->partattrs, fromkey->partattrs,
> +                            newkey->partnatts * sizeof(AttrNumber));
>
> It's wasteful to use palloc0 if you're immediately going to overwrite
> every byte in the array.  Use regular palloc instead.

Done.

> +    /* Only this can ever be NULL */
> +    if (!partexprbinDatum)
> +    {
> +        nulls[Anum_pg_partitioned_table_partexprbin - 1] = true;
> +        nulls[Anum_pg_partitioned_table_partexprsrc - 1] = true;
> +    }
>
> How can it be valid to have no partitioning expressions?

Keys that are simply column names are resolved to attnums and stored
likewise.  If some key is an expression, then corresponding attnum is 0
and the expression itself is added to the list that gets stored into
partexprbin.  It is doing the same thing as index expressions.

> +    /* Tell world about the key */
> +    CacheInvalidateRelcache(rel);
>
> Is this really needed?  Isn't the caller going to do something similar
> pretty soon?

Oops, I removed that in one of the later patches but shouldn't be there in
the first place.  Fixed.

> +    heap_freetuple(tuple);
>
> Probably useless - might as well let the context reset clean it up.

Removed.

> +    simple_heap_delete(rel, &tuple->t_self);
> +
> +    /* Update the indexes on pg_partitioned_table */
> +    CatalogUpdateIndexes(rel, tuple);
>
> You don't need CatalogUpdateIndexes() after a delete, only after an
> insert or update.

Ah right, removed.

>
> +    if (classform->relkind != relkind &&
> +                (relkind == RELKIND_RELATION &&
> +                    classform->relkind != RELKIND_PARTITIONED_TABLE))
>
> That's broken.  Note that all of the conditions are joined using &&,
> so if any one of them fails then we won't throw an error.  In
> particular, it's no longer possible to throw an error when relkind is
> not RELKIND_RELATION.

You are right.  I guess it would have to be the following:

+    if ((classform->relkind != relkind &&
+         classform->relkind != RELKIND_PARTITIONED_TABLE) ||
+        (classform->relkind == RELKIND_PARTITIONED_TABLE &&
+         relkind != RELKIND_RELATION))

Such hackishness could not be helped because we have a separate DROP
command for every distinct relkind, except we overload DROP TABLE for both
regular and partitioned tables.

> +/* Checks if a Var node is for a given attnum */
> +static bool
> +find_attr_reference_walker(Node *node, find_attr_reference_context *context)
> +{
> +    if (node == NULL)
> +        return false;
> +
> +    if (IsA(node, Var))
> +    {
> +        Var       *var = (Var *) node;
> +        char   *varattname = get_attname(context->relid, var->varattno);
> +
> +        if (!strcmp(varattname, context->attname))
> +            return true;
> +    }
> +
> +    return expression_tree_walker(node, find_attr_reference_walker, context);
> +}
>
> Hrm.  The comment says we're matching on attnum, but the code says
> we're matching on attname.  is_partition_attr() has the same confusion
> between comments and code.  Maybe instead of this whole approach it
> would be better to use pull_varattnos(), then get_attnum() to find the
> attribute number for the one you want, then bms_is_member().

I like the idea to use pull_varattnos(), so done that way.

> +static PartitionBy *
> +transformPartitionBy(Relation rel, PartitionBy *partitionby)
> +{
> +    PartitionBy       *partby;
> +    ParseState       *pstate;
> +    RangeTblEntry  *rte;
> +    ListCell       *l;
> +
> +    partby = (PartitionBy *) makeNode(PartitionBy);
> +
> +    partby->strategy = partitionby->strategy;
> +    partby->location = partitionby->location;
> +    partby->partParams = NIL;
> +
> +    /*
> +     * Create a dummy ParseState and insert the target relation as its sole
> +     * rangetable entry.  We need a ParseState for transformExpr.
> +     */
> +    pstate = make_parsestate(NULL);
>
> Why isn't this logic being invoked from transformCreateStmt()?  Then
> we could use the actual parseState for the query instead of a fake
> one.

Because we need an open relation for it to work, which in this case there
won't be until after we have performed heap_create_with_catalog() in
DefineRelation().  Mainly because we need to perform transformExpr() on
expressions.  That's similar to how cookConstraint() on the new CHECK
constraints cannot be performed earlier.  Am I missing something?

> +            if (IsA(expr, CollateExpr))
> +                ereport(ERROR,
> +                        (errcode(ERRCODE_INVALID_OBJECT_DEFINITION),
> +                         errmsg("cannot use COLLATE in partition key
> expression")));
>
> I assume there is a good reason for this seemingly-arbitrary
> restriction, but there's no comment saying what it is.  One thing
> that's odd is that this will only prohibit a CollateExpr at the top
> level, not in some more-deeply nested position.  That seems
> suspicious.

Hmm, I guess it wouldn't hurt to just leave any COLLATE clauses as it is -
just remove the above code.  Of course, we must then record the collation
in the catalog alongside other user-specified information such as operator
class.  Currently, if the key is a simple column we use its attcollation
and if it's an expression then we use its exprCollation().

When I first wrote it, I wasn't sure what the implications of explicit
collations would be for partitioning.  There are two places where it comes
into play: a) when comparing partition key values using the btree compare
function b) embedded as varcollid and inputcollid in implicitly generated
check constraints for partitions.  In case of the latter, any mismatch
with query-specified collation causes constraint exclusion proof to be
canceled.  When it's default collations everywhere, the chances of that
sort of thing happening are less.  If we support user-specified collations
on keys, then things will get a little bit more involved.

Thoughts?

> +                /*
> +                 * User wrote "(column)" or "(column COLLATE something)".
> +                 * Treat it like simple attribute anyway.
> +                 */
>
> Evidently, the user did not do that, because you just prohibited the
> second one of those.

Holds true now that I have removed the prohibition.

> +                if (IsA(expr, Const))
> +                    ereport(ERROR,
> +                            (errcode(ERRCODE_INVALID_OBJECT_DEFINITION),
> +                             errmsg("cannot use a constant expression
> as partition key")));
> +
> +                if (contain_mutable_functions(expr))
> +                    ereport(ERROR,
> +                            (errcode(ERRCODE_INVALID_OBJECT_DEFINITION),
> +                             errmsg("functions in partition key
> expression must be marked IMMUTABLE")));
>
> Do these checks parallel what we do for CHECK constraints?  It might
> be good to apply about the same level of rigor in both cases.

Both of these checks are not done for CHECK constraints.  The only check
performed on them in cookConstraint() checks whether the expression is of
boolean type.

However in the present case, this is just one side of a whole partition
constraint (the other piece being individual partition's bound value), so
should be treated a bit differently from the CHECK constraints.  I modeled
this on ComputeIndexAttrs() checks viz. the following:

/*
 * An expression using mutable functions is probably wrong,
 * since if you aren't going to get the same result for the
 * same data every time, it's not clear what the index entries
 * mean at all.
 */
 if (CheckMutability((Expr *) expr))
     ereport(ERROR,
         (errcode(ERRCODE_INVALID_OBJECT_DEFINITION),
          errmsg("functions in index expression must be marked IMMUTABLE")));

Likewise if a partition key expression contained mutable functions, same
input row could be mapped to different partitions based on the result of
expression computed using the input values.  So, it seems prudent to
enforce immutability unlike CHECK constraints.  Am I missing something?

>
> +                exprsrc = deparse_expression(expr,
> +                            deparse_context_for(RelationGetRelationName(rel),
> +                                                RelationGetRelid(rel)),
> +                                       false, false);
>
> Why bother?  The output of this doesn't seem like a useful thing to
> store.  The fact that we've done similar things elsewhere doesn't make
> it a good idea.  I think we did it in other cases because we used to
> be dumber than we are now.

Deparsed expressions are no longer stored in the catalog.

> +                        (errcode(ERRCODE_UNDEFINED_OBJECT),
> +                         errmsg("data type %s has no default btree
> operator class",
> +                                format_type_be(atttype)),
> +                         errhint("You must specify an existing btree
> operator class or define one for the type.")));
>
> The hint is not really accurate, because the type may well have a
> btree operator class.  Just not a default one.

Changed to: "You must specify a btree operator class or define a default
btree operator class for the data type."

> +                            char    relkind = ((CreateStmt *)
> stmt)->partby != NULL
> +                                                    ? RELKIND_PARTITIONED_TABLE
> +                                                    : RELKIND_RELATION;
>
> Let's push this down into DefineRelation().  i.e. if (stmt->partby !=
> NULL) { if (relkind != RELKIND_RELATION) ereport(...); relkind =
> RELKIND_PARTITION_TABLE; }

Done.  By the way, I made the ereport say the following: "unexpected
relkind value passed to DefineRelation", did you intend it to  say
something else?

>
> +        RelationBuildPartitionKey(relation);
>
> I wonder if RelationBuildPartitionKey should really be in relcache.c.
> What do we do in similar cases?

There a number of RelationBuild* functions that RelationBuildDesc calls
that are reside outside relcache.c in respective modules - trigger.c
(BuildTrigger), policy.c (BuildRowSecurity), whereas some others such as
RelationInitIndexAccessInfo() are housed in relcache.c.

BuildPartitionKey used to be in relcache.c and can be moved back there. So
did.

>
> +} PartitionBy;
>
> Maybe PartitionSpec?

A later patch uses PartitionListSpec and PartitionRangeSpec as parse nodes
for partition bounds.  How about call PartitionBy PartitionKeySpec and the
bound nodes just mentioned PartitionBoundList and PartitionBoundRange,
respectively?


Attached revised patches.  In addition to addressing comments in this
email (which addressed only the patch 0001), number of other fixes [1] and
other miscellaneous improvements [2] have been included.

Thanks,
Amit

[1]
https://www.postgresql.org/message-id/2896956a-8af0-3f91-a3bc-1e5225ae7ed2@lab.ntt.co.jp

[2]
https://www.postgresql.org/message-id/98d6c7e7-4919-4970-a158-49f740101812%40lab.ntt.co.jp

On Thu, Sep 29, 2016 at 8:09 AM, Amit Langote
<Langote_Amit_f8@lab.ntt.co.jp> wrote:
> I removed DEPENDENCY_IGNORE.  Does the following look good or am I still
> missing something?

You missed your commit message, but otherwise looks fine.

>> Also, I think this should be rephrased a bit to be more clear about
>> how the partitioning key works, like this: The optional
>> <literal>PARTITION BY</literal> clause specifies a method of
>> partitioning the table.  The table thus created is called a
>> <firstterm>partitioned</firstterm> table.  The parenthesized list of
>> expressions forms the <firsttem>partitioning key</firstterm> for the
>> table.  When using range partitioning, the partioning key can include
>> multiple columns or expressions, but for list partitioning, the
>> partitioning key must consist of a single column or expression.  If no
>> btree operator class is specified when creating a partitioned table,
>> the default btree operator class for the datatype will be used.  If
>> there is none, an error will be reported.
>
> Revised text along these lines.

It seems you copied my typo: my text has "partioning" for
"partitioning" and your patch now has that, too.

> Things were that way initially, that is, the parent relations had no
> relfilenode.  I abandoned that project however.  The storage-less parent
> thing seemed pretty daunting to me to handle right away.  For example,
> optimizer and the executor code needed to be taught about the parent rel
> appendrel member that used to be included as part of the result of
> scanning the inheritance set but was no longer.

Even if we leave the empty relfilenode around for now -- in the long
run I think it should die -- I think we should prohibit the creation
of subsidiary object on the parent which is only sensible if it has
rows - e.g. indexes.  It makes no sense to disallow non-inheritable
constraints while allowing indexes, and it could box us into a corner
later.

>> +        /*
>> +         * Run the expressions through eval_const_expressions. This is
>> +         * not just an optimization, but is necessary, because eventually
>> +         * the planner will be comparing them to similarly-processed qual
>> +         * clauses, and may fail to detect valid matches without this.
>> +         * We don't bother with canonicalize_qual, however.
>> +         */
>>
>> I'm a bit confused by this, because I would think this processing
>> ought to have been done before storing anything in the system
>> catalogs.  I don't see why it should be necessary to do it again after
>> pulling data back out of the system catalogs.
>
> The pattern matches what's done for other expressions that optimizer deals
> with, such as CHECK, index key, and index predicate expressions.

That's kind of a non-answer answer, but OK.  Still, I think you
shouldn't just copy somebody else's comment blindly into a new place.
Reference the original comment, or write your own.

>> How can it be valid to have no partitioning expressions?
>
> Keys that are simply column names are resolved to attnums and stored
> likewise.  If some key is an expression, then corresponding attnum is 0
> and the expression itself is added to the list that gets stored into
> partexprbin.  It is doing the same thing as index expressions.

Oh, right.  Oops.

>> +    if (classform->relkind != relkind &&
>> +                (relkind == RELKIND_RELATION &&
>> +                    classform->relkind != RELKIND_PARTITIONED_TABLE))
>>
>> That's broken.  Note that all of the conditions are joined using &&,
>> so if any one of them fails then we won't throw an error.  In
>> particular, it's no longer possible to throw an error when relkind is
>> not RELKIND_RELATION.
>
> You are right.  I guess it would have to be the following:
>
> +    if ((classform->relkind != relkind &&
> +         classform->relkind != RELKIND_PARTITIONED_TABLE) ||
> +        (classform->relkind == RELKIND_PARTITIONED_TABLE &&
> +         relkind != RELKIND_RELATION))
>
> Such hackishness could not be helped because we have a separate DROP
> command for every distinct relkind, except we overload DROP TABLE for both
> regular and partitioned tables.

Maybe this would be better:

if (relkind == RELKIND_PARTITIONED_TABLE)   syntax_relkind = RELKIND_RELATION;
else   syntax_relkind = rekind;
if (classform->relkind != syntax_relkind)   DropErrorMsgWrongType(rel->relname, classform->relkind, relkind);

>> Why isn't this logic being invoked from transformCreateStmt()?  Then
>> we could use the actual parseState for the query instead of a fake
>> one.
>
> Because we need an open relation for it to work, which in this case there
> won't be until after we have performed heap_create_with_catalog() in
> DefineRelation().  Mainly because we need to perform transformExpr() on
> expressions.  That's similar to how cookConstraint() on the new CHECK
> constraints cannot be performed earlier.  Am I missing something?

Hmm, yeah, I guess that's the same thing.  I guess I got on this line
of thinking because the function name started with "transform" which
is usually something happening during parse analysis only.  Maybe add
a code comment explaining why the work can't be done sooner, just like
what you've written here.

> Hmm, I guess it wouldn't hurt to just leave any COLLATE clauses as it is -
> just remove the above code.  Of course, we must then record the collation
> in the catalog alongside other user-specified information such as operator
> class.  Currently, if the key is a simple column we use its attcollation
> and if it's an expression then we use its exprCollation().
>
> When I first wrote it, I wasn't sure what the implications of explicit
> collations would be for partitioning.  There are two places where it comes
> into play: a) when comparing partition key values using the btree compare
> function b) embedded as varcollid and inputcollid in implicitly generated
> check constraints for partitions.  In case of the latter, any mismatch
> with query-specified collation causes constraint exclusion proof to be
> canceled.  When it's default collations everywhere, the chances of that
> sort of thing happening are less.  If we support user-specified collations
> on keys, then things will get a little bit more involved.

I think it's worth rewinding to general principles here: the only time
a collation has real meaning is in the context of a comparison
operation.  If we ask whether A < B, the answer may depend on the
collation that is used to perform the comparison.  Any other place
that mentions a collation is only trying to influence the collation
that gets used for some comparison that will happen later - so, for
example, for a table column, attcollation is setting the default
collation for comparisons involving that column.  The column is not
itself collated; that doesn't really make sense.

In the case of partitioning, there is really exactly one thing that
matters: everybody's got to agree on the collation to be used to
compare actual or hypothetical values of the partitioning columns
against the partition bounds.  If we've got a set of partition bounds
b_1, b_2, ..., b_n and a set of partitions p_1, p_2, ..., p_{n-1} such
that a row in p_i must have a key k such that b_i < k and k < b_{i+1},
and if the meaning of that "<" operator is collation-dependent, then
we had better agree on which collation is in use every single time we
do the test.

Indexes, of course, have this exact same problem, at least if, like
btree, they rely on ordering.  We search the index by applying the "<"
operator to compare various values taken from the index tuples with
the value we're trying to find, and it is absolutely vital that the
collation used for lookups remain constant and that it is the same as
the collation used for inserts, which had also better remain constant.
That is why pg_index has an indcollation column of type oidvector: it
tells us which collation to use for each index column.  It pairs with
indclass, which tells us which operator class to use for each index
column.  I think that partitioning will need exact analogues -
partclass and partcollation - because it has exactly the same problem.
Currently, you've got partclass but not partcollation.

I'd in general recommend that you try to follow the index precedent
here as closely as practical.

> However in the present case, this is just one side of a whole partition
> constraint (the other piece being individual partition's bound value), so
> should be treated a bit differently from the CHECK constraints.  I modeled
> this on ComputeIndexAttrs() checks viz. the following:
>
> /*
>  * An expression using mutable functions is probably wrong,
>  * since if you aren't going to get the same result for the
>  * same data every time, it's not clear what the index entries
>  * mean at all.
>  */
>  if (CheckMutability((Expr *) expr))
>      ereport(ERROR,
>          (errcode(ERRCODE_INVALID_OBJECT_DEFINITION),
>           errmsg("functions in index expression must be marked IMMUTABLE")));
>
> Likewise if a partition key expression contained mutable functions, same
> input row could be mapped to different partitions based on the result of
> expression computed using the input values.  So, it seems prudent to
> enforce immutability unlike CHECK constraints.  Am I missing something?

OK, seems reasonable.

>> +                            char    relkind = ((CreateStmt *)
>> stmt)->partby != NULL
>> +                                                    ? RELKIND_PARTITIONED_TABLE
>> +                                                    : RELKIND_RELATION;
>>
>> Let's push this down into DefineRelation().  i.e. if (stmt->partby !=
>> NULL) { if (relkind != RELKIND_RELATION) ereport(...); relkind =
>> RELKIND_PARTITION_TABLE; }
>
> Done.  By the way, I made the ereport say the following: "unexpected
> relkind value passed to DefineRelation", did you intend it to  say
> something else?

You don't need it to be translatable because it should only happen if
there's a bug in the code; users shouldn't actually be able to hit it.
So use elog() rather than ereport().  You don't need to include the
function name, either; the user can use \errverbose or \set VERBOSITY
verbose to get the file and line number if they need it.  So just
elog(ERROR, "unexpected relkind") should be fine; or maybe elog(ERROR,
"unexpected relkind: %d", (int) relkind), following similar precedent
elsewhere.

Reading through the latest 0001:

+   The catalog <structname>pg_partitioned_table</structname> stores information
+   about the partition key of tables.

Maybe "stores information about how tables are partitioned".

+       Partitioning strategy (or method); <literal>l</> = list
partitioned table,

I don't think we need "(or method)".

+      <entry><structfield>partexprbin</structfield></entry>

Is there any good reason not to do partexprbin -> partexpr throughout the patch?

+      A partitioned table is divided into sub-tables (called partitions), which
+      in turn, are created using separate <literal>CREATE TABLE</> commands.

I would delete "in turn" and the subsequent comma, so that it says
"which are created using separate".

+      The table itself is empty.  A data row inserted into the table is mapped
+      to and stored in one of the partitions (if one exists) based on the
+      values of columns or expressions in the partition key and partition
+      bound constraints associated with the individual partitions.

How about: "A data row inserted into the table is routed to a
partition based on the value of columns or expressions in the
partition key.  If no existing partition matches the values in the new
row, an error will occur."

+      Partitioned tables do not support UNIQUE, PRIMARY, EXCLUDE, or FOREIGN
+      KEY constraints; however, you can define these constraints on individual
+      data partitions.

Delete "data".   Add <literal> tags around the keywords.

+    tuple = SearchSysCache1(PARTEDRELID,
+                            ObjectIdGetDatum(RelationGetRelid(rel)));
+    /* Cannot already exist */
+    Assert(!HeapTupleIsValid(tuple));

This seems pointless.  It's hard to see how the tuple could already
exist, but if it does, this will fail a little later on anyway when we
do simple_heap_insert() and CatalogUpdateIndexes() for the new tuple.
In general, if you're doing work only to support an Assert(), you
should put #ifdef USE_ASSERT_CHECKING around it, but in this case I'd
just rip this out.

+    myself.objectId = RelationGetRelid(rel);;

Extra semicolon.

+    /*
+     * Store dependencies on anything mentioned in the key expressions.
+     * However, ignore the column references which causes self-dependencies
+     * to be created that are undesirable.  That is done by asking the
+     * dependency-tracking sub-system to ignore any such dependencies.
+     */

I think this comment is spending a lot of time explaining the
mechanism when what it should be doing is explaining why this case
arises here and not elsewhere.

+    if (relkind != RELKIND_RELATION  && relkind != RELKIND_PARTITIONED_TABLE)

Extra space before &&.

+        if(partattno != 0)

Missing space.

+         * Identify a btree opclass to use. Currently, we use only btree
+         * operators which seems enough for list and range partitioning.

Probably best to add a comma before "which".
                break;            case T_ForeignKeyCacheInfo:                _outForeignKeyCacheInfo(str, obj);
+            case T_PartitionSpec:
+                _outPartitionSpec(str, obj);
+                break;
+            case T_PartitionElem:
+                _outPartitionElem(str, obj);                break;
            default:

Missing break.

+                    n->strategy = PARTITION_STRAT_RANGE;

Let's not abbreviate STRATEGY to STRAT in the names of these constants.
        case EXPR_KIND_TRIGGER_WHEN:            return "WHEN";
+        case EXPR_KIND_PARTITION_KEY:
+            return "partition key expression";

I think you should say "PARTITION BY" here.  See the function header
comment for ParseExprKindName.

+                 errmsg("cannot use more than one column in partition key"),
+                 errdetail("Only one column allowed with list
partitioning.")));

How about combining these two: cannot list partition using more than one column

+ * Note that the partition key data attached to a relcache entry must be
+ * stored CacheMemoryContext to ensure it survives as long as the relcache
+ * entry. But we should be running in a less long-lived working context.
+ * To avoid leaking cache memory if this routine fails partway through,
+ * we build in working memory and then copy the completed structure into
+ * cache memory.

Again, don't just copy existing comments.  Refer to them.

+     * To retrieve further variable-length attributes, we'd need the catlog's

Typo.

+ * pg_partitioned_table_fn.h
+ *      prototypes for functions in catalog/pg_partitioned_table.c

Is it really worth having a separate header file for ONE function?

+PG_KEYWORD("list", LIST, UNRESERVED_KEYWORD)

I bet you can avoid making this a keyword.  PARTITION BY IDENT in the
grammar, or something like that.

+CREATE TABLE pkrel(
+    a int PRIMARY KEY
+);

Add a space.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Fri, Sep 30, 2016 at 9:10 AM, Robert Haas <robertmhaas@gmail.com> wrote:
> On Thu, Sep 29, 2016 at 8:09 AM, Amit Langote
> <Langote_Amit_f8@lab.ntt.co.jp> wrote:
>> I removed DEPENDENCY_IGNORE.  Does the following look good or am I still
>> missing something?
>
> You missed your commit message, but otherwise looks fine.

I have moved this patch to next CF because that's fresh, but switched
the patch as "waiting on author". Be careful, the patch was in "needs
review".
-- 
Michael

On 2016/10/03 13:26, Michael Paquier wrote:
> On Fri, Sep 30, 2016 at 9:10 AM, Robert Haas <robertmhaas@gmail.com> wrote:
>> On Thu, Sep 29, 2016 at 8:09 AM, Amit Langote
>> <Langote_Amit_f8@lab.ntt.co.jp> wrote:
>>> I removed DEPENDENCY_IGNORE.  Does the following look good or am I still
>>> missing something?
>>
>> You missed your commit message, but otherwise looks fine.
> 
> I have moved this patch to next CF because that's fresh, but switched
> the patch as "waiting on author". Be careful, the patch was in "needs
> review".

Thanks Michael.  I was going to post the patch addressing Robert's
comments today, which I will anyway.  Then I will switch the status back
to "Needs Review", albeit as part of the next CF.

Thanks,
Amit

Thanks for the comments.

On 2016/09/30 9:10, Robert Haas wrote:
> On Thu, Sep 29, 2016 at 8:09 AM, Amit Langote
> <Langote_Amit_f8@lab.ntt.co.jp> wrote:
>> Things were that way initially, that is, the parent relations had no
>> relfilenode.  I abandoned that project however.  The storage-less parent
>> thing seemed pretty daunting to me to handle right away.  For example,
>> optimizer and the executor code needed to be taught about the parent rel
>> appendrel member that used to be included as part of the result of
>> scanning the inheritance set but was no longer.
>
> Even if we leave the empty relfilenode around for now -- in the long
> run I think it should die -- I think we should prohibit the creation
> of subsidiary object on the parent which is only sensible if it has
> rows - e.g. indexes.  It makes no sense to disallow non-inheritable
> constraints while allowing indexes, and it could box us into a corner
> later.

I agree.  So we must prevent from the get-go the creation of following
objects on parent tables (aka RELKIND_PARTITIONED_TABLE relations):

* Indexes
* Row triggers (?)

In addition to preventing creation of these objects, we must also teach
commands that directly invoke heapam.c to skip such relations.

I have not implemented that in the patch yet though.

>>> +        /*
>>> +         * Run the expressions through eval_const_expressions. This is
>>> +         * not just an optimization, but is necessary, because eventually
>>> +         * the planner will be comparing them to similarly-processed qual
>>> +         * clauses, and may fail to detect valid matches without this.
>>> +         * We don't bother with canonicalize_qual, however.
>>> +         */
>>>
>>> I'm a bit confused by this, because I would think this processing
>>> ought to have been done before storing anything in the system
>>> catalogs.  I don't see why it should be necessary to do it again after
>>> pulling data back out of the system catalogs.
>>
>> The pattern matches what's done for other expressions that optimizer deals
>> with, such as CHECK, index key, and index predicate expressions.
>
> That's kind of a non-answer answer, but OK.  Still, I think you
> shouldn't just copy somebody else's comment blindly into a new place.
> Reference the original comment, or write your own.

Sorry I could have explained a bit more in my previous message.  I rewrote
the comment as follows:

/*
 * Run the expressions through const-simplification since the planner
 * will be comparing them to similarly-processed qual clause operands,
 * and may fail to detect valid matches without this step.  We don't
 * need to bother with canonicalize_qual() though, because partition
 * expressions are not full-fledged qualification clauses.
 */

>>> +    if (classform->relkind != relkind &&
>>> +                (relkind == RELKIND_RELATION &&
>>> +                    classform->relkind != RELKIND_PARTITIONED_TABLE))
>>>
>>> That's broken.  Note that all of the conditions are joined using &&,
>>> so if any one of them fails then we won't throw an error.  In
>>> particular, it's no longer possible to throw an error when relkind is
>>> not RELKIND_RELATION.
>>
>> You are right.  I guess it would have to be the following:
>>
>> +    if ((classform->relkind != relkind &&
>> +         classform->relkind != RELKIND_PARTITIONED_TABLE) ||
>> +        (classform->relkind == RELKIND_PARTITIONED_TABLE &&
>> +         relkind != RELKIND_RELATION))
>>
>> Such hackishness could not be helped because we have a separate DROP
>> command for every distinct relkind, except we overload DROP TABLE for both
>> regular and partitioned tables.
>
> Maybe this would be better:
>
> if (relkind == RELKIND_PARTITIONED_TABLE)
>     syntax_relkind = RELKIND_RELATION;
> else
>     syntax_relkind = rekind;
> if (classform->relkind != syntax_relkind)
>     DropErrorMsgWrongType(rel->relname, classform->relkind, relkind);

In this case, relkind refers to the command viz. DROP <ObjectType/relkind>
<relname>.  It can never be RELKIND_PARTITIONED_TABLE, so the above will
be equivalent to leaving things unchanged.  Anyway I agree the suggested
style is better, but I assume you meant:

    if (classform->relkind == RELKIND_PARTITIONED_TABLE)
        expected_relkind = RELKIND_RELATION;
    else
        expected_relkind = classform->relkind;

    if (relkind != expected_relkind)
        DropErrorMsgWrongType(rel->relname, classform->relkind, relkind);


>>> Why isn't this logic being invoked from transformCreateStmt()?  Then
>>> we could use the actual parseState for the query instead of a fake
>>> one.
>>
>> Because we need an open relation for it to work, which in this case there
>> won't be until after we have performed heap_create_with_catalog() in
>> DefineRelation().  Mainly because we need to perform transformExpr() on
>> expressions.  That's similar to how cookConstraint() on the new CHECK
>> constraints cannot be performed earlier.  Am I missing something?
>
> Hmm, yeah, I guess that's the same thing.  I guess I got on this line
> of thinking because the function name started with "transform" which
> is usually something happening during parse analysis only.  Maybe add
> a code comment explaining why the work can't be done sooner, just like
> what you've written here.

Added the comment.

>> Hmm, I guess it wouldn't hurt to just leave any COLLATE clauses as it is -
>> just remove the above code.  Of course, we must then record the collation
>> in the catalog alongside other user-specified information such as operator
>> class.  Currently, if the key is a simple column we use its attcollation
>> and if it's an expression then we use its exprCollation().
>>
>> When I first wrote it, I wasn't sure what the implications of explicit
>> collations would be for partitioning.  There are two places where it comes
>> into play: a) when comparing partition key values using the btree compare
>> function b) embedded as varcollid and inputcollid in implicitly generated
>> check constraints for partitions.  In case of the latter, any mismatch
>> with query-specified collation causes constraint exclusion proof to be
>> canceled.  When it's default collations everywhere, the chances of that
>> sort of thing happening are less.  If we support user-specified collations
>> on keys, then things will get a little bit more involved.
>
> I think it's worth rewinding to general principles here: the only time
> a collation has real meaning is in the context of a comparison
> operation.  If we ask whether A < B, the answer may depend on the
> collation that is used to perform the comparison.  Any other place
> that mentions a collation is only trying to influence the collation
> that gets used for some comparison that will happen later - so, for
> example, for a table column, attcollation is setting the default
> collation for comparisons involving that column.  The column is not
> itself collated; that doesn't really make sense.
>
> In the case of partitioning, there is really exactly one thing that
> matters: everybody's got to agree on the collation to be used to
> compare actual or hypothetical values of the partitioning columns
> against the partition bounds.  If we've got a set of partition bounds
> b_1, b_2, ..., b_n and a set of partitions p_1, p_2, ..., p_{n-1} such
> that a row in p_i must have a key k such that b_i < k and k < b_{i+1},
> and if the meaning of that "<" operator is collation-dependent, then
> we had better agree on which collation is in use every single time we
> do the test.
>
> Indexes, of course, have this exact same problem, at least if, like
> btree, they rely on ordering.  We search the index by applying the "<"
> operator to compare various values taken from the index tuples with
> the value we're trying to find, and it is absolutely vital that the
> collation used for lookups remain constant and that it is the same as
> the collation used for inserts, which had also better remain constant.
> That is why pg_index has an indcollation column of type oidvector: it
> tells us which collation to use for each index column.  It pairs with
> indclass, which tells us which operator class to use for each index
> column.  I think that partitioning will need exact analogues -
> partclass and partcollation - because it has exactly the same problem.
> Currently, you've got partclass but not partcollation.
>
> I'd in general recommend that you try to follow the index precedent
> here as closely as practical.

Thanks for the explanation. I added a new field to the catalog called
partcollation.

That means a couple of things - when comparing input key values (consider
a new tuple being routed) to partition bounds using the btree compare
function, we use the corresponding key column's partcollation.  The same
is also used as inputcollid of the OpExpr (or ScalarArrayOpExpr) in the
implicitly generated CHECK constraints.

However, varcollid of any Var nodes in the above expressions is still the
corresponding attributes' attcollation.

Needless to say, a query-specified qualification clause must now include
the same collate clause as used for individual partition columns (or
expressions) for the constraint exclusion to work as intended.

>>> +                            char    relkind = ((CreateStmt *)
>>> stmt)->partby != NULL
>>> +                                                    ? RELKIND_PARTITIONED_TABLE
>>> +                                                    : RELKIND_RELATION;
>>>
>>> Let's push this down into DefineRelation().  i.e. if (stmt->partby !=
>>> NULL) { if (relkind != RELKIND_RELATION) ereport(...); relkind =
>>> RELKIND_PARTITION_TABLE; }
>>
>> Done.  By the way, I made the ereport say the following: "unexpected
>> relkind value passed to DefineRelation", did you intend it to  say
>> something else?
>
> You don't need it to be translatable because it should only happen if
> there's a bug in the code; users shouldn't actually be able to hit it.
> So use elog() rather than ereport().  You don't need to include the
> function name, either; the user can use \errverbose or \set VERBOSITY
> verbose to get the file and line number if they need it.  So just
> elog(ERROR, "unexpected relkind") should be fine; or maybe elog(ERROR,
> "unexpected relkind: %d", (int) relkind), following similar precedent
> elsewhere.

Done as: elog(ERROR, "unexpected relkind: %d", (int) relkind)

> Reading through the latest 0001:
>
> +   The catalog <structname>pg_partitioned_table</structname> stores information
> +   about the partition key of tables.
>
> Maybe "stores information about how tables are partitioned".

Fixed.

> +       Partitioning strategy (or method); <literal>l</> = list
> partitioned table,
>
> I don't think we need "(or method)".

Fixed.

>
> +      <entry><structfield>partexprbin</structfield></entry>
>
> Is there any good reason not to do partexprbin -> partexpr throughout the patch?

Agreed. Though I used partexprs (like indexprs).

> +      A partitioned table is divided into sub-tables (called partitions), which
> +      in turn, are created using separate <literal>CREATE TABLE</> commands.
>
> I would delete "in turn" and the subsequent comma, so that it says
> "which are created using separate".

Done.

>
> +      The table itself is empty.  A data row inserted into the table is mapped
> +      to and stored in one of the partitions (if one exists) based on the
> +      values of columns or expressions in the partition key and partition
> +      bound constraints associated with the individual partitions.
>
> How about: "A data row inserted into the table is routed to a
> partition based on the value of columns or expressions in the
> partition key.  If no existing partition matches the values in the new
> row, an error will occur."

Done.

>
> +      Partitioned tables do not support UNIQUE, PRIMARY, EXCLUDE, or FOREIGN
> +      KEY constraints; however, you can define these constraints on individual
> +      data partitions.
>
> Delete "data".   Add <literal> tags around the keywords.

Done.

> +    tuple = SearchSysCache1(PARTEDRELID,
> +                            ObjectIdGetDatum(RelationGetRelid(rel)));
> +    /* Cannot already exist */
> +    Assert(!HeapTupleIsValid(tuple));
>
> This seems pointless.  It's hard to see how the tuple could already
> exist, but if it does, this will fail a little later on anyway when we
> do simple_heap_insert() and CatalogUpdateIndexes() for the new tuple.
> In general, if you're doing work only to support an Assert(), you
> should put #ifdef USE_ASSERT_CHECKING around it, but in this case I'd
> just rip this out.

OK, removed the Assert.

> +    myself.objectId = RelationGetRelid(rel);;
>
> Extra semicolon.

Fixed.

> +    /*
> +     * Store dependencies on anything mentioned in the key expressions.
> +     * However, ignore the column references which causes self-dependencies
> +     * to be created that are undesirable.  That is done by asking the
> +     * dependency-tracking sub-system to ignore any such dependencies.
> +     */
>
> I think this comment is spending a lot of time explaining the
> mechanism when what it should be doing is explaining why this case
> arises here and not elsewhere.

Rewrote the comment as follows:

/*
 * Anything mentioned in the expressions.  We must ignore the column
 * references which will count as self-dependency items; in this case,
 * the depender is the table itself (there is no such thing as partition
 * key object).
 */

> +    if (relkind != RELKIND_RELATION  && relkind != RELKIND_PARTITIONED_TABLE)
>
> Extra space before &&.

Fixed.

>
> +        if(partattno != 0)
>
> Missing space.

Fixed.

>
> +         * Identify a btree opclass to use. Currently, we use only btree
> +         * operators which seems enough for list and range partitioning.
>
> Probably best to add a comma before "which".

Done.

>
>                  break;
>              case T_ForeignKeyCacheInfo:
>                  _outForeignKeyCacheInfo(str, obj);
> +            case T_PartitionSpec:
> +                _outPartitionSpec(str, obj);
> +                break;
> +            case T_PartitionElem:
> +                _outPartitionElem(str, obj);
>                  break;
>
>              default:
>
> Missing break.

Oops, fixed.

> +                    n->strategy = PARTITION_STRAT_RANGE;
>
> Let's not abbreviate STRATEGY to STRAT in the names of these constants.

Done.

>
>          case EXPR_KIND_TRIGGER_WHEN:
>              return "WHEN";
> +        case EXPR_KIND_PARTITION_KEY:
> +            return "partition key expression";
>
> I think you should say "PARTITION BY" here.  See the function header
> comment for ParseExprKindName.

Used "PARTITION BY".  I was trying to mimic EXPR_KIND_INDEX_EXPRESSION,
but this seems to work better.  Also, I renamed EXPR_KIND_PARTITION_KEY to
EXPR_KIND_PARTITION_EXPRESSION.

By the way, a bunch of error messages say "partition key expression".  I'm
assuming it's better to leave them alone, that is, not rewrite them as
"PARTITION BY expressions".

> +                 errmsg("cannot use more than one column in partition key"),
> +                 errdetail("Only one column allowed with list
> partitioning.")));
>
> How about combining these two: cannot list partition using more than one column

Done.

> + * Note that the partition key data attached to a relcache entry must be
> + * stored CacheMemoryContext to ensure it survives as long as the relcache
> + * entry. But we should be running in a less long-lived working context.
> + * To avoid leaking cache memory if this routine fails partway through,
> + * we build in working memory and then copy the completed structure into
> + * cache memory.
>
> Again, don't just copy existing comments.  Refer to them.

Rewrote the comment to explain two points: why build in the working
context instead of directly in the CacheMemoryContext and why use a
private child context of CacheMemoryContext to store the information.

> +     * To retrieve further variable-length attributes, we'd need the catlog's
>
> Typo.

Fixed.

> + * pg_partitioned_table_fn.h
> + *      prototypes for functions in catalog/pg_partitioned_table.c
>
> Is it really worth having a separate header file for ONE function?

Previously the two functions in this file were defined in catalog/heap.c
and I think they could be moved back there alongside such folks as
AddRelationNewConstraints, StoreAttrDefault, RemoveAttributeById,
RemoveStatistics, etc.  So did.

There are no longer pg_partitioned_table.c and pg_partitioned_table_fn.h
files.

> +PG_KEYWORD("list", LIST, UNRESERVED_KEYWORD)
>
> I bet you can avoid making this a keyword.  PARTITION BY IDENT in the
> grammar, or something like that.

Managed to do that with one more production as follows:

+part_strategy:  IDENT                   { $$ = $1; }
+                | unreserved_keyword    { $$ = pstrdup($1); }
+        ;

And then PARTITION BY part_strategy

That's because "range" is already a keyword.

>
> +CREATE TABLE pkrel(
> +    a int PRIMARY KEY
> +);
>
> Add a space.

Fixed.


Attached updated patches.

Thanks,
Amit

On Tue, Oct 4, 2016 at 4:02 AM, Amit Langote
<Langote_Amit_f8@lab.ntt.co.jp> wrote:
>> Even if we leave the empty relfilenode around for now -- in the long
>> run I think it should die -- I think we should prohibit the creation
>> of subsidiary object on the parent which is only sensible if it has
>> rows - e.g. indexes.  It makes no sense to disallow non-inheritable
>> constraints while allowing indexes, and it could box us into a corner
>> later.
>
> I agree.  So we must prevent from the get-go the creation of following
> objects on parent tables (aka RELKIND_PARTITIONED_TABLE relations):
>
> * Indexes
> * Row triggers (?)

Hmm, do we ever fire triggers on the parent for operations on a child
table?  Note this thread, which seems possibly relevant:

https://www.postgresql.org/message-id/flat/cd282adde5b70b20c57f53bb9ab75e27%40biglumber.com

We certainly won't fire the parent's per-row triggers ever, so those
should be prohibited.  But I'm not sure about per-statement triggers.

> In addition to preventing creation of these objects, we must also teach
> commands that directly invoke heapam.c to skip such relations.

I'm don't think that's required if we're not actually getting rid of
the relfilenode.

> In this case, relkind refers to the command viz. DROP <ObjectType/relkind>
> <relname>.  It can never be RELKIND_PARTITIONED_TABLE, so the above will
> be equivalent to leaving things unchanged.  Anyway I agree the suggested
> style is better, but I assume you meant:
>
>     if (classform->relkind == RELKIND_PARTITIONED_TABLE)
>         expected_relkind = RELKIND_RELATION;
>     else
>         expected_relkind = classform->relkind;
>
>     if (relkind != expected_relkind)
>         DropErrorMsgWrongType(rel->relname, classform->relkind, relkind);

Uh, yeah, probably that's what I meant. :-)

> Thanks for the explanation. I added a new field to the catalog called
> partcollation.
>
> That means a couple of things - when comparing input key values (consider
> a new tuple being routed) to partition bounds using the btree compare
> function, we use the corresponding key column's partcollation.  The same
> is also used as inputcollid of the OpExpr (or ScalarArrayOpExpr) in the
> implicitly generated CHECK constraints.

Check.

> However, varcollid of any Var nodes in the above expressions is still the
> corresponding attributes' attcollation.

I think that's OK.

> Needless to say, a query-specified qualification clause must now include
> the same collate clause as used for individual partition columns (or
> expressions) for the constraint exclusion to work as intended.

Hopefully this is only required if the default choice of collation
wouldn't be correct anyway.

>> +      <entry><structfield>partexprbin</structfield></entry>
>>
>> Is there any good reason not to do partexprbin -> partexpr throughout the patch?
>
> Agreed. Though I used partexprs (like indexprs).

Oh, good idea.

>>          case EXPR_KIND_TRIGGER_WHEN:
>>              return "WHEN";
>> +        case EXPR_KIND_PARTITION_KEY:
>> +            return "partition key expression";
>>
>> I think you should say "PARTITION BY" here.  See the function header
>> comment for ParseExprKindName.
>
> Used "PARTITION BY".  I was trying to mimic EXPR_KIND_INDEX_EXPRESSION,
> but this seems to work better.  Also, I renamed EXPR_KIND_PARTITION_KEY to
> EXPR_KIND_PARTITION_EXPRESSION.
>
> By the way, a bunch of error messages say "partition key expression".  I'm
> assuming it's better to leave them alone, that is, not rewrite them as
> "PARTITION BY expressions".

I think that is fine.  ParseExprKindName is something of a special case.

Reviewing 0002:

+    prettyFlags = PRETTYFLAG_INDENT;
+    PG_RETURN_TEXT_P(string_to_text(pg_get_partkeydef_worker(relid,
+                                    prettyFlags)));

Why bother with the variable?

+    /* Must get partclass, and partexprs the hard way */
+    datum = SysCacheGetAttr(PARTEDRELID, tuple,
+                            Anum_pg_partitioned_table_partclass, &isnull);
+    Assert(!isnull);
+    partclass = (oidvector *) DatumGetPointer(datum);

Comment mentions getting two things, but code only gets one.

+        partexprs = (List *) stringToNode(exprsString);

if (!IsA(partexprs, List)) elog(ERROR, ...); so as to guard against
corrupt catalog contents.

+    switch (form->partstrat)
+    {
+        case 'l':
+            appendStringInfo(&buf, "LIST");
+            break;
+        case 'r':
+            appendStringInfo(&buf, "RANGE");
+            break;
+    }

default: elog(ERROR, "unexpected partition strategy: %d", (int)
form->partstrat);

Also, why not case PARTITION_STRATEGY_LIST: instead of case 'l': and
similarly for 'r'?

@@ -5296,7 +5301,8 @@ getTables(Archive *fout, int *numTables)                          "OR %s IS NOT NULL "
             "OR %s IS NOT NULL"                          "))"
 
-                          "AS changed_acl "
+                          "AS changed_acl, "
+                          "CASE WHEN c.relkind = 'P' THEN
pg_catalog.pg_get_partkeydef(c.oid) ELSE NULL END AS partkeydef "                          "FROM pg_class c "
              "LEFT JOIN pg_depend d ON "                          "(c.relkind = '%c' AND "
 

I think if you test it you'll find this breaks dumps from older server
versions.  You've got to add a new version of the SQL query for 10+,
and then update the 9.6, 9.5, 9.4, 9.3, 9.1-9.2, 9.0, 8.4, 8.2-8.3,
8.0-8.1, 7.3-7.4, 7.2, 7.1, and pre-7.1 queries to return a dummy NULL
value for that column. Alternatively, you can add the new version for
10.0, leave the older queries alone, and then adjust the code further
down to cope with i_partkeydef == -1 (see i_checkoption for an
example).
                      gettext_noop("table"),
+                      gettext_noop("table"),

Add a comment like /* partitioned table */ on the same line.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Tue, Oct 4, 2016 at 4:02 AM, Amit Langote
<Langote_Amit_f8@lab.ntt.co.jp> wrote:
> [ latest patch set ]

Reviewing 0003:

+      This form attaches an existing table (partitioned or otherwise) as

(which might itself be partitioned)

+      partition of the target table.  Partition bound specification must
+      correspond with the partition method and the key of the target table.

The partition bound specification must correspond to the partitioning
method and partitioning key of the target table.

+      The table being attached must have all the columns of the target table
+      with matching types and no more. Also, it must have all the matching

The table to be attached must have all of the same columns as the
target table and no more; moreover, the column types must also match.

+      with matching types and no more. Also, it must have all the matching
+      constraints as the target table.  That includes both <literal>NOT NULL</>
+      and <literal>CHECK</> constraints.  If some <literal>CHECK</literal>
+      constraint of the table being attached is marked <literal>NO
INHERIT</literal>,
+      the command will fail; such constraints must either be dropped or
+      recreated without the <literal>NO INHERIT</literal> mark.

Why all of these requirements?  We could instead perform a scan to
validate that the constraints are met.  I think the way this should
work is:

1. ATTACH PARTITION works whether matching NOT NULL and CHECK
constraints are present or not.

2. If all of the constraints are present, and a validated constraint
matching the implicit partitioning constraint is also present, then
ATTACH PARTITION does not scan the table to validate constraints;
otherwise, it does.

3. NO VALIDATE is not an option.

+      Currently <literal>UNIQUE</literal>, <literal>PRIMARY KEY</literal>, and
+      <literal>FOREIGN KEY</literal> constraints are not considered, but that
+      might change in the future.

Really?  Changing that sounds impractical to me.

+      This form detaches specified partition of the target table.  The
+      detached partition continues to exist as a standalone table with no ties
+      remaining with the target table.

continues to exist as a standalone table, but no longer has any ties
to the table from which it was detached.

+      Note that if a partition being detached is itself a partitioned table,
+      it continues to exist as such.

You don't really need to say this, I think.  All of the properties of
the detached table are retained, not only its partitioning status.
You wouldn't like it if I told you to document "note that if a
partition being detached is unlogged, it will still be unlogged".
   To add the table as a new child of a parent table, you must own the
-   parent table as well.
+   parent table as well.  That applies to both adding the table as a
+   inheritance child of a parent table and attaching a table as partition to
+   the table.

To add the table as a new child of a parent table, or as a new
partition of a partitioned table, you must own the parent table as
well.

+        The name of the table to attach as a new partition to or
detach from this table.

s/to or/or to/

+    <literal>NO VALIDATE</> option is spcified.

Typo, but see comments above about nuking this altogether.
    A recursive <literal>DROP COLUMN</literal> operation will remove a    descendant table's column only if the
descendantdoes not inherit    that column from any other parents and never had an independent
 
-    definition of the column.  A nonrecursive <literal>DROP
+    definition of the column (which always holds if the descendant table
+    is a partition).  A nonrecursive <literal>DROP    COLUMN</literal> (i.e., <command>ALTER TABLE ONLY ... DROP
COLUMN</command>)never removes any descendant columns, but
 
-    instead marks them as independently defined rather than inherited.
+    instead marks them as independently defined rather than inherited,
+    unless the descendant table is a partition.

This is a hairy explanation.  I suggest that the documentation say
this (and that the code implement it): A nonrecursive DROP TABLE
command will fail for a partitioned table, because all partitions of a
table must have the same columns as the partitioning root.

-    that are not marked <literal>NO INHERIT</>.
+    that are not marked <literal>NO INHERIT</> which are unsupported if
+    the table is a partitioned table.

I think you can omit this hunk.

+   If <literal>PARTITION OF</literal> clause is specified then the table is
+   created as a partition of <literal>parent_table</literal> with specified
+   bounds.  However, unlike regular tables, one cannot specify
+   <literal>PARTITION BY</literal> clause which means foreign tables can
+   only be created as leaf partitions.

I'd delete the sentence beginning with "However".

+   Create foreign table <structname>measurement_y2016m07</>, which will be
+   accessed through the server <structname>server_07</>, that is partition
+   of the range partitioned table <structname>measurement</>:

s/, that is/ as a/

+<phrase>and <replaceable
class="PARAMETER">partition_bound_spec</replaceable> is:</phrase>
+
+FOR VALUES { <replaceable class="PARAMETER">list_spec</replaceable> |
<replaceable class="PARAMETER">range_spec</replaceable> }

I think you can inline the definitions of list_spec and range_spec
here instead of making them separate productions, and I think that
would be preferable.

FOR VALUES { IN ( <replaceable
class="PARAMETER">expression</replaceable> [, ...] ) |
START <replaceable class="PARAMETER">lower-bound</replaceable> [
INCLUSIVE | EXCLUSIVE ] END <replaceable
class="PARAMETER">upper-bound</replaceable> [ INCLUSIVE | EXCLUSIVE ]
}

+      parent table (name optionally schema-qualified).

Parenthetical phrase isn't needed.

+      A partition bound specification must be present and must correspond with
+      partition method and key of the parent table.  It is checked using the
+      specification that the new partition does not overlap with any existing
+      partitions of the parent.

The partition bound specification must correspond to the partitioning
method and partitioning key of the parent table, and must not overlap
with any existing partition of that parent.

+      clause, if any.  Defaults and constraints can optionally be specified
+      for each of the inherited columns, which override those in the parent.

Surely not.  You can't "override" an inherited constraint or an
inherited default.  The child may have extra constraints not present
in the parent, and may have different defaults when it is directly
targeted by an insertion, but it can't possibly override the parent
defaults.

+      One can also specify table constraints, in addition to those inherited
+      from the parent.  Note that all subsequent schema modifications to the
+      parent propagate to partition.

The first part of this seems right, but then what's going on with the
reference to constraints in the previous sentence?  (See previous
review comment.) The second sentence I would delete (but see below).

+     <para>
+      Any data row subsequently inserted into the parent table is mapped to
+      and stored in the partition, provided partition key of the row falls
+      within the partition bounds.
+     </para>

How about: Rows inserted into a partitioned table will be
automatically routed to the correct partition.  If no suitable
partition exists, an error will occur.

+     <para>
+      A partition is dropped or truncated when the parent table is dropped or
+      truncated.  Dropping it directly using <literal>DROP TABLE</literal>
+      will fail; it must first be <firstterm>detached</> from the parent.
+      However, truncating a partition directly works.
+     </para>

How about: A partition must have the same column names and types as
the table of which it is a partition.  Therefore, modifications the
column names or types of the partitioned table will automatically
propagate to all children, as will operations such as TRUNCATE which
normally affect a table and all of its inheritance children.  It is
also possible to TRUNCATE a partition individually, just as for an
inheritance child.

Insisting that you can't drop a child without detaching it first seems
wrong to me.  If I already made this comment and you responded to it,
please point me back to whatever you said.  However, my feeling is
this is flat wrong and absolutely must be changed.

-            if (is_no_inherit)
+
+            /* Discard the NO INHERIT flag if the relation is a partition */
+            if (is_no_inherit && !rel->rd_rel->relispartition)

Something about this seems fishy.  In StoreRelCheck(), you disallow
the addition of a NO INHERIT constraint on a partition, but here you
seem happy to accept it and ignore the NO INHERIT property.  That
doesn't seem consistent.  We should adopt a consistent policy about
what to do about such constraints, and I submit that throwing an error
is better than silently changing things, unless you have some reason
for doing that which I'm not seeing.  Anyway, we should have the same
behavior in both cases.

We should also standardize on what value of conislocal and coninhcount
children end up with; presumably the latter should be 1, but I'm not
sure if the former should be true or false.  In either case, anything
that can vary between children probably needs to be dumped, so let's
enforce that it doesn't so we don't have to dump it.  I'm not sure
whether the code here achieves those objectives, though, and note the
comment in the function header about making sure the logic here
matches MergeConstraintsIntoExisting.

I think the overriding principle here should be: If you attach a table
as a partition, it must not be part of a standard inheritance
hierarchy, and it must not be a partition of any other table.  It can,
however, be partitioned itself.  If you later detach a partition, it
ends up as a standalone table with a copy of each constraint it had as
a partition - probably including the implicit partition constraint.
The DBA can drop those constraints if they're not wanted.

I wonder if it's really a good idea for the partition constraints to
be implicit; what is the benefit of leaving those uncatalogued?

+ * Depending on whether the relation in question is list or range
+ * partitioned, one of the fields is set.
+ */
+typedef struct BoundCollectionData
+{
+    struct ListInfo       *listinfo;
+    struct RangeInfo   *rangeinfo;
+} BoundCollectionData;

This seems like an odd design.  First, when you have a pointer to
either of two things, the normal tool for that in C would be a union,
not a struct. Second, in PostgreSQL we typically handle that by making
both of the things nodes and then you can use IsA() or switch on
nodeTag() to figure out what you've got.  Third, the only place this
is used at least in 0003 is as part of PartitionDescData, which only
has 3 members, so if you were going to do it with 2 members, you could
just include these two members directly.  Considering all of the
foregoing, I'd suggest making this a union and including partstrategy
in PartitionDescData.

I think that the names ListInfo and RangeInfo are far too generic for
something that's specific to partitioning.

+/*
+ * Range bound collection - sorted array of ranges of partitions of a range
+ * partitioned table
+ */
+typedef struct RangeInfo
+{
+    struct PartitionRange    **ranges;
+} RangeInfo;
+
+/* One partition's range */
+typedef struct PartitionRange
+{
+    struct PartitionRangeBound    *lower;
+    struct PartitionRangeBound    *upper;
+} PartitionRange;

This representation doesn't seem optimal, because in most cases the
lower bound of one partition will be the upper bound of the next.  I
suggest that you flatten this representation into a single list of
relevant bounds, each flagged as to whether it is exclusive and
whether it is finite; and a list with one more element of bounds.  For
example, suppose the partition bounds are [10, 20), [20, 30), (30,
40), and [50, 60).  You first construct a list of all of the distinct
bounds, flipping inclusive/exclusive for the lefthand bound in each
case.  So you get:

10 EXCLUSIVE
20 EXCLUSIVE
30 EXCLUSIVE
30 INCLUSIVE
40 EXCLUSIVE
50 EXCLUSIVE
60 EXCLUSIVE

When ordering items for this list, if the same item appears twice, the
EXCLUSIVE copy always appears before INCLUSIVE.  When comparing
against an EXCLUSIVE item, we move to the first half of the array if
we are searching for a value strictly less than that value; when
comparing against an INCLUSIVE item, we move to the first half of the
array if we are searching for a value less than or equal to that
value.

This is a list of seven items, so a binary search will return a
position between 0 (less than all items) and 7 (greater than all
items).  So we need a list of 8 partition mappings, which in this case
will look like this: -1, 0, 1, -1, 2, -1, 3, -1.

In this particular example, there are only two adjacent partitions, so
we end up with 7 bounds with this representation vs. 8 with yours, but
in general I think the gains will be bigger.  If you've got 1000
partitions and they're all adjacent, which is likely, you store 1000
bounds instead of 2000 bounds by doing it this way.

+ * Note: This function should be called only when it is known that 'relid'
+ * is a partition.

Why?  How about "Because this function assumes that the relation whose
OID is passed as an argument will have precisely one parent, it should
only been called when it is known that the relation is a partition."

+    /*
+     * Translate vars in the generated expression to have correct attnos.
+     * Note that the vars in my_qual bear attnos dictated by key which carries
+     * physical attnos of the parent.  We must allow for a case where physical
+     * attnos of a partition can be different from the parent.
+     */
+    partexprs_item = list_head(key->partexprs);
+    for (i = 0; i < key->partnatts; i++)
+    {
+        AttrNumber    attno = key->partattrs[i],
+                    new_attno;
+        char       *attname;
+
+        if (attno != 0)
+        {
+            /* Simple column reference */
+            attname = get_attname(RelationGetRelid(parent), attno);
+            new_attno = get_attnum(RelationGetRelid(rel), attname);
+
+            if (new_attno != attno)
+                my_qual = (List *) translate_var_attno((Node *) my_qual,
+                                                       attno,
+                                                       new_attno);

It can't really be safe to do this one attribute number at a time, or
even if by some good fortune it can't be broken, it at least it seems
extremely fragile.  Suppose that you translate 0 -> 3 and then 3 -> 5;
now the result is garbage.  It's not very efficient to do this one
attno at a time, either.

+    if (classform->relispartition)
+        ereport(ERROR,
+                (errcode(ERRCODE_WRONG_OBJECT_TYPE),
+                 errmsg("\"%s\" is a partition of \"%s\"", rel->relname,
+                        get_rel_name(get_partition_parent(relOid))),
+                 errhint("Use ALTER TABLE DETACH PARTITION to be able
to drop it.")));
+

RangeVarCallbackForDropRelation should do only minimal sanity checks;
defer this until after we have a relation lock.

I didn't get all the way through this patch, so this is a pretty
incomplete review, but it's late here and I'm out of steam for
tonight.  Some general comments:

1. I think that this patch seems to introduce an awful lot of new
structures with confusingly similar names and purposes:
PartitionBoundList, PartitionBoundRange, ListInfo, RangeInfo,
PartitionRange, PartitionList, ListValue, RangePartition.  You've got
4 different structures here that all have "Partition" and "Range" in
the name someplace, including both PartitionRange and RangePartition.
Maybe there's no way around that kind of duplication; after all there
are quite a few moving parts here.  But it seems like it would be good
to try to simplify it.

2. I'm also a bit concerned about the fairly large amount of
apparently-boilerplate code in partition.c, all having to do with how
we create all of these data structures and translate between different
forms of them.  I haven't understood that stuff thoroughly enough to
have a specific idea about how we might be able to get rid of any of
it, and maybe there's no way.  But that too seems like a topic for
futher investigation.  One idea is that maybe some of these things
should be nodes that piggyback on the existing infrastructure in
src/backend/nodes instead of inventing a new way to do something
similar.

3. There are a lot of places where you have separate code for the
range and list partitioning cases, and I'm suspicious that there are
ways that code could be unified.  For example, with list partitioning,
you have a bunch of Datums which represent the specific values that
can appear in the various partitions, and with range partitioning, you
have a bunch of Datums that represent the edges of partitions.  Well,
if you used the same array for both purposes, you could use the same
code to copy it.  That would involve flattening away a lot of the
subsidiary structure under PartitionDescData and pulling up stuff that
is buried lower down into the main structure, but I think that's
likely a good idea anyway - see also point #1.

4. I'm somewhat wondering if we ought to just legislate that the lower
bound is always inclusive and the upper bound is always exclusive.
The decision to support both inclusive and exclusive partition bounds
is responsible for an enormous truckload of complexity in this patch,
and I have a feeling it is also going to be a not-infrequent cause of
user error.

5. I wonder how well this handles multi-column partition keys.  You've
just got one Datum flag and one is-finite flag per partition, but I
wonder if you don't need to track is-finite on a per-column basis, so
that you could partition on (a, b) and have the first partition go up
to (10, 10), the second to (10, infinity), the third to (20, 10), the
fourth to (20, infinity), and the last to (infinity, infinity).  FWIW,
Oracle supports this sort of thing, so perhaps we should, too.  On a
related note, I'm not sure it's going to work to treat a composite
partition key as a record type.  The user may want to specify a
separate opfamily and collation for each column, not just inherit
whatever the record behavior is.  I'm not sure if that's what you are
doing, but the relcache structures don't seem adapted to storing one
Datum per partitioning column per partition, but rather just one Datum
per partition, and I'm not sure that's going to work very well.

Thanks for working on this.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 05/10/16 03:50, Robert Haas wrote:
> On Tue, Oct 4, 2016 at 4:02 AM, Amit Langote
> <Langote_Amit_f8@lab.ntt.co.jp> wrote:
>> [ latest patch set ]
> 
> Reviewing 0003:
> 
> +      with matching types and no more. Also, it must have all the matching
> +      constraints as the target table.  That includes both <literal>NOT NULL</>
> +      and <literal>CHECK</> constraints.  If some <literal>CHECK</literal>
> +      constraint of the table being attached is marked <literal>NO
> INHERIT</literal>,
> +      the command will fail; such constraints must either be dropped or
> +      recreated without the <literal>NO INHERIT</literal> mark.
> 
> Why all of these requirements?  We could instead perform a scan to
> validate that the constraints are met.  I think the way this should
> work is:

I think it's survivable limitation in initial version, it does not seem
to me like there is anything that prevents improving this in some follow
up patch.

> 
> +      Currently <literal>UNIQUE</literal>, <literal>PRIMARY KEY</literal>, and
> +      <literal>FOREIGN KEY</literal> constraints are not considered, but that
> +      might change in the future.
> 
> Really?  Changing that sounds impractical to me.
> 

Which part of that statement?

> 
> +      Note that if a partition being detached is itself a partitioned table,
> +      it continues to exist as such.
> 
> You don't really need to say this, I think.  All of the properties of
> the detached table are retained, not only its partitioning status.
> You wouldn't like it if I told you to document "note that if a
> partition being detached is unlogged, it will still be unlogged".
> 

I think this is bit different though, it basically means you are
detaching whole branch from the rest of the partitioning tree, not just
that one partition. To me that's worth mentioning to avoid potential
confusion.

--  Petr Jelinek                  http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training &
Services

<div dir="ltr"><br /><div class="gmail_extra">On Tue, Oct 4, 2016 at 1:32 PM, Amit Langote <span dir="ltr"><<a
href="mailto:Langote_Amit_f8@lab.ntt.co.jp"target="_blank">Langote_Amit_f8@lab.ntt.co.jp</a><wbr />></span>
wrote:<span><br/><br /></span><span></span><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px
0px0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span> </span>Attached updated patches.<br /><br
/>Thanks,<br /> Amit<br /></blockquote></div><br />Hi,<br /><br />I observed, when creating foreign table with range
partition,data is not inserting into specified partition range. below are steps to reproduce.<br /><br />CREATE
EXTENSIONpostgres_fdw;<br />CREATE SERVER pwj_server FOREIGN DATA WRAPPER postgres_fdw OPTIONS (dbname 'postgres', port
'5432',use_remote_estimate'true');<br />CREATE USER MAPPING FOR PUBLIC SERVER pwj_server;<br /><br />CREATE TABLE
test_range(a int) PARTITION BY RANGE(a);<br /><br />CREATE TABLE test_range_p1 (a int);<br />CREATE FOREIGN TABLE
ft_test_range_p1PARTITION OF test_range FOR VALUES START (1) END (10) SERVER pwj_server OPTIONS (TABLE_NAME
'test_range_p1');<br/><br />CREATE TABLE test_range_p2 (a int);<br />CREATE FOREIGN TABLE ft_test_range_p2 PARTITION OF
test_rangeFOR VALUES START (20) END (30) SERVER pwj_server OPTIONS (TABLE_NAME 'test_range_p2');<br /><br />CREATE
TABLEtest_range_p3 (a int);<br />CREATE FOREIGN TABLE ft_test_range_p3 PARTITION OF test_range FOR VALUES START (10)
END(20) SERVER pwj_server OPTIONS (TABLE_NAME 'test_range_p3');<br /><br />postgres=# INSERT INTO test_range (a) values
(5),(25),(15);<br/>INSERT 0 3<br /><br />postgres=# select tableoid::regclass, * from test_range;<br />    
tableoid    | a  <br />------------------+----<br /> ft_test_range_p1 |  5<br /> ft_test_range_p2 | 15<br
/> ft_test_range_p3| 25<br />(3 rows)<br /><br /></div><div class="gmail_extra">--Here ft_test_range_p2 is created for
range20-30 having value 15.<br /></div><div class="gmail_extra"><br />Thanks & Regards,<br />Rajkumar
Raghuwanshi<br/>QMG, EnterpriseDB Corporation<br /><br /></div></div> 

Hi,

On 2016/10/05 16:57, Rajkumar Raghuwanshi wrote:
> I observed, when creating foreign table with range partition, data is not
> inserting into specified partition range. below are steps to reproduce.
> 
> [ ... ]
> 
> postgres=# INSERT INTO test_range (a) values (5),(25),(15);
> INSERT 0 3
> 
> postgres=# select tableoid::regclass, * from test_range;
>      tableoid     | a 
> ------------------+----
>  ft_test_range_p1 |  5
>  ft_test_range_p2 | 15
>  ft_test_range_p3 | 25
> (3 rows)
> 
> --Here ft_test_range_p2 is created for range 20-30 having value 15.

Thanks a lot for testing.

That's a bug.  I found that it is caused by the FDW plans getting paired
with the wrong result relations during ExecInitModifyTable() initialization.

I will include a fix for the same in the patch set that I will be sending
soon in reply to Robert's review comments on patch 0002 [1].

Thanks,
Amit

[1]
https://www.postgresql.org/message-id/CA%2BTgmoY1aQ5iPz0S2GBJw4YUR1Z2Qg5iKUf8YJSo2Ctya4ZmNg%40mail.gmail.com

On 2016/10/05 2:12, Robert Haas wrote:
> On Tue, Oct 4, 2016 at 4:02 AM, Amit Langote
> <Langote_Amit_f8@lab.ntt.co.jp> wrote:
>>> Even if we leave the empty relfilenode around for now -- in the long
>>> run I think it should die -- I think we should prohibit the creation
>>> of subsidiary object on the parent which is only sensible if it has
>>> rows - e.g. indexes.  It makes no sense to disallow non-inheritable
>>> constraints while allowing indexes, and it could box us into a corner
>>> later.
>>
>> I agree.  So we must prevent from the get-go the creation of following
>> objects on parent tables (aka RELKIND_PARTITIONED_TABLE relations):
>>
>> * Indexes
>> * Row triggers (?)
>
> Hmm, do we ever fire triggers on the parent for operations on a child
> table?  Note this thread, which seems possibly relevant:
>
> https://www.postgresql.org/message-id/flat/cd282adde5b70b20c57f53bb9ab75e27%40biglumber.com

The answer to your question is no.

The thread you quoted discusses statement-level triggers and the
conclusion is that they don't work as desired for UPDATE and DELETE on
inheritance tables.  As things stand, only UPDATE or DELETE on the parent
affects the child tables and it's proposed there that the statement-level
triggers on the parent and also on any child tables affected should be
fired in that case.

Currently, INSERT doesn't have that problem because it only ever affects
the parent table.  That changes with tuple routing though.  I am not sure
if the aforementioned proposed behavior should be applied in this case.
That is to say, if we have INSERT INTO parent, only the parent's
statement-level triggers should be fired.  The partition into which the
tuple is routed will have only its row-level triggers fired.

The proposed TODO item has been left untouched for many years now, so it
is perhaps not such a big pain point after all (or perhaps some standard
noncompliance).  I may be wrong though.

> We certainly won't fire the parent's per-row triggers ever, so those
> should be prohibited.  But I'm not sure about per-statement triggers.

Agree about the row-level triggers.  I think per-statement triggers are
alright to allows on parent tables from the POV of the original topic we
were discussing - ie, per-statement triggers do not depend on the physical
addressing capabilities of the relation on which they are defined whereas
row-levels triggers do in some cases.

So I changed 0001 so that indexes and row triggers are disallowed on
partitioned tables.

>> In addition to preventing creation of these objects, we must also teach
>> commands that directly invoke heapam.c to skip such relations.
>
> I'm don't think that's required if we're not actually getting rid of
> the relfilenode.

Oops that line got sent after all, I had convinced myself to remove that
line from the email. :)  Agreed, no need to do that just yet.

>> Needless to say, a query-specified qualification clause must now include
>> the same collate clause as used for individual partition columns (or
>> expressions) for the constraint exclusion to work as intended.
>
> Hopefully this is only required if the default choice of collation
> wouldn't be correct anyway.

Sorry, I'm not sure I understand what you said - specifically what would
"wrong" mean in this case? Parser error?  Run-time error like in varstr_cmp()?

What I meant to say is that predicate OpExpr's generated by the
partitioning code will be using the corresponding partition column's
partcollation as its inputcollid (and the Var will be wrapped by a
RelabelType).  Here partcollation is a known valid alternative collation
that is applied when partitioning the input data, overriding any default
collation of the column's type or the collation specified for the column
when creating the table.  During assign_query_collations(), a qual clause
OpExpr's variable will get assigned a collation that is either the
column/expressions's attcollation/exprCollationor explicitly specified
collation using a COLLATE clause.  If variable collation assigned thusly
doesn't match the partitioning collation, then the predicate OpExpr's and
the query clause OpExpr's variable expressions will fail to match using
equal(), so constraint exclusion (specifically, operator_predicate_proof()
bails out) will fail.  Of course, going ahead with constraint exclusion
ignoring the said collation mismatch would be wrong anyway.

> Reviewing 0002:
>
> +    prettyFlags = PRETTYFLAG_INDENT;
> +    PG_RETURN_TEXT_P(string_to_text(pg_get_partkeydef_worker(relid,
> +                                    prettyFlags)));
>
> Why bother with the variable?

Leftover, removed.

> +    /* Must get partclass, and partexprs the hard way */
> +    datum = SysCacheGetAttr(PARTEDRELID, tuple,
> +                            Anum_pg_partitioned_table_partclass, &isnull);
> +    Assert(!isnull);
> +    partclass = (oidvector *) DatumGetPointer(datum);
>
> Comment mentions getting two things, but code only gets one.

Fixed. Further code uses the "hard way" to get partexprs but it's got a
separate comment anyway.

>
> +        partexprs = (List *) stringToNode(exprsString);
>
> if (!IsA(partexprs, List)) elog(ERROR, ...); so as to guard against
> corrupt catalog contents.

Done as follows:

    partexprs = (List *) stringToNode(exprsString);

    if (!IsA(partexprs, List))
        elog(ERROR, "unexpected node type found in partexprs: %d",
                    (int) nodeTag(partexprs));

> +    switch (form->partstrat)
> +    {
> +        case 'l':
> +            appendStringInfo(&buf, "LIST");
> +            break;
> +        case 'r':
> +            appendStringInfo(&buf, "RANGE");
> +            break;
> +    }
>
> default: elog(ERROR, "unexpected partition strategy: %d", (int)
> form->partstrat);
>
> Also, why not case PARTITION_STRATEGY_LIST: instead of case 'l': and
> similarly for 'r'?

Done and done.

> @@ -5296,7 +5301,8 @@ getTables(Archive *fout, int *numTables)
>                            "OR %s IS NOT NULL "
>                            "OR %s IS NOT NULL"
>                            "))"
> -                          "AS changed_acl "
> +                          "AS changed_acl, "
> +                          "CASE WHEN c.relkind = 'P' THEN
> pg_catalog.pg_get_partkeydef(c.oid) ELSE NULL END AS partkeydef "
>                            "FROM pg_class c "
>                            "LEFT JOIN pg_depend d ON "
>                            "(c.relkind = '%c' AND "
>
> I think if you test it you'll find this breaks dumps from older server
> versions.  You've got to add a new version of the SQL query for 10+,
> and then update the 9.6, 9.5, 9.4, 9.3, 9.1-9.2, 9.0, 8.4, 8.2-8.3,
> 8.0-8.1, 7.3-7.4, 7.2, 7.1, and pre-7.1 queries to return a dummy NULL
> value for that column. Alternatively, you can add the new version for
> 10.0, leave the older queries alone, and then adjust the code further
> down to cope with i_partkeydef == -1 (see i_checkoption for an
> example).

You're quite right.  Fixed using this last method.

>                        gettext_noop("table"),
> +                      gettext_noop("table"),
>
> Add a comment like /* partitioned table */ on the same line.

Done.

Attached revised patches.  Also, includes a fix for an issue reported by
Rajkumar Raghuwanshi [1] which turned out to be a bug in one of the later
patches.  I will now move on to addressing the comments on patch 0003.

Thanks a lot for the review!

Thanks,
Amit

[1]
https://www.postgresql.org/message-id/5dded2f1-c7f6-e7fc-56b5-23ab59495e4b@lab.ntt.co.jp

On 2016/10/07 17:33, Rajkumar Raghuwanshi wrote:
> On Thu, Oct 6, 2016 at 12:44 PM, Amit Langote wrote:
> I have applied latest patches, getting some error and crash, please check
> if you are also able to reproduce the same.
>
> Observation1 : Not able to create index on partition table.
> --------------
> CREATE TABLE rp (c1 int, c2 int) PARTITION BY RANGE(c1);
> CREATE TABLE rp_p1 PARTITION OF rp FOR VALUES START (1) END (10);
> CREATE TABLE rp_p2 PARTITION OF rp FOR VALUES START (10) END (20);
>
> CREATE INDEX idx_rp_c1 on rp(c1);
> ERROR:  cannot create index on partitioned table "rp"

This one is a new behavior as I mentioned earlier in my previous email.

> Observation2 : Getting cache lookup failed error for multiple column range
> partition
> --------------
> CREATE TABLE rp1_m (c1 int, c2 int) PARTITION BY RANGE(c1, ((c1 + c2)/2));
>
> CREATE TABLE rp1_m_p1 PARTITION OF rp1_m FOR VALUES START (1, 1) END (10,
> 10);
> ERROR:  cache lookup failed for attribute 0 of relation 16429
>
> Observation3 : Getting server crash with multiple column range partition
> --------------
> CREATE TABLE rp2_m (c1 int, c2 int) PARTITION BY RANGE(((c2 + c1)/2), c2);
> CREATE TABLE rp2_m_p1 PARTITION OF rp2_m FOR VALUES START (1, 1) END (10,
> 10);
> server closed the connection unexpectedly
>     This probably means the server terminated abnormally
>     before or while processing the request.
> The connection to the server was lost. Attempting reset: Failed.
> !>

Fixed.  Should have been caught when running the regression tests after I
made changes to 0001 to address some review comments (apparently there was
no test in 0003 that would've caught this, so added a new one, thanks!).


Attached updated patches.

Thanks,
Amit

It's allowed to specify an non-default opclass in partition by clause,
but I do not see any testcase testing the same. Can you please add
one.

On Fri, Oct 7, 2016 at 2:50 PM, Amit Langote
<Langote_Amit_f8@lab.ntt.co.jp> wrote:
> On 2016/10/07 17:33, Rajkumar Raghuwanshi wrote:
>> On Thu, Oct 6, 2016 at 12:44 PM, Amit Langote wrote:
>> I have applied latest patches, getting some error and crash, please check
>> if you are also able to reproduce the same.
>>
>> Observation1 : Not able to create index on partition table.
>> --------------
>> CREATE TABLE rp (c1 int, c2 int) PARTITION BY RANGE(c1);
>> CREATE TABLE rp_p1 PARTITION OF rp FOR VALUES START (1) END (10);
>> CREATE TABLE rp_p2 PARTITION OF rp FOR VALUES START (10) END (20);
>>
>> CREATE INDEX idx_rp_c1 on rp(c1);
>> ERROR:  cannot create index on partitioned table "rp"
>
> This one is a new behavior as I mentioned earlier in my previous email.
>
>> Observation2 : Getting cache lookup failed error for multiple column range
>> partition
>> --------------
>> CREATE TABLE rp1_m (c1 int, c2 int) PARTITION BY RANGE(c1, ((c1 + c2)/2));
>>
>> CREATE TABLE rp1_m_p1 PARTITION OF rp1_m FOR VALUES START (1, 1) END (10,
>> 10);
>> ERROR:  cache lookup failed for attribute 0 of relation 16429
>>
>> Observation3 : Getting server crash with multiple column range partition
>> --------------
>> CREATE TABLE rp2_m (c1 int, c2 int) PARTITION BY RANGE(((c2 + c1)/2), c2);
>> CREATE TABLE rp2_m_p1 PARTITION OF rp2_m FOR VALUES START (1, 1) END (10,
>> 10);
>> server closed the connection unexpectedly
>>     This probably means the server terminated abnormally
>>     before or while processing the request.
>> The connection to the server was lost. Attempting reset: Failed.
>> !>
>
> Fixed.  Should have been caught when running the regression tests after I
> made changes to 0001 to address some review comments (apparently there was
> no test in 0003 that would've caught this, so added a new one, thanks!).
>
>
> Attached updated patches.
>
> Thanks,
> Amit



-- 
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company

On 2016/10/07 18:27, Ashutosh Bapat wrote:
> It's allowed to specify an non-default opclass in partition by clause,
> but I do not see any testcase testing the same. Can you please add
> one.

OK, I will add some tests related to that.

Thanks,
Amit

On Thu, Oct 6, 2016 at 12:44 PM, Amit Langote
<Langote_Amit_f8@lab.ntt.co.jp> wrote:
> On 2016/10/05 2:12, Robert Haas wrote:
>> On Tue, Oct 4, 2016 at 4:02 AM, Amit Langote
>> <Langote_Amit_f8@lab.ntt.co.jp> wrote:
>>>> Even if we leave the empty relfilenode around for now -- in the long
>>>> run I think it should die -- I think we should prohibit the creation
>>>> of subsidiary object on the parent which is only sensible if it has
>>>> rows - e.g. indexes.  It makes no sense to disallow non-inheritable
>>>> constraints while allowing indexes, and it could box us into a corner
>>>> later.
>>>
>>> I agree.  So we must prevent from the get-go the creation of following
>>> objects on parent tables (aka RELKIND_PARTITIONED_TABLE relations):
>>>
>>> * Indexes
>>> * Row triggers (?)
>>
>> Hmm, do we ever fire triggers on the parent for operations on a child
>> table?  Note this thread, which seems possibly relevant:
>>
>> https://www.postgresql.org/message-id/flat/cd282adde5b70b20c57f53bb9ab75e27%40biglumber.com
>
> The answer to your question is no.
>
> The thread you quoted discusses statement-level triggers and the
> conclusion is that they don't work as desired for UPDATE and DELETE on
> inheritance tables.  As things stand, only UPDATE or DELETE on the parent
> affects the child tables and it's proposed there that the statement-level
> triggers on the parent and also on any child tables affected should be
> fired in that case.
>

Doesn't that imply that the statement level triggers should be fired
for all the tables that get changed for statement?  If so, then in
your case it should never fire for parent table, which means we could
disallow statement level triggers as well on parent tables?

Some of the other things that we might want to consider disallowing on
parent table could be:
a. Policy on table_name
b. Alter table has many clauses, are all of those allowed and will it
make sense to allow them?

-- 
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

On Thu, Oct 6, 2016 at 12:44 PM, Amit Langote
<Langote_Amit_f8@lab.ntt.co.jp> wrote:
> On 2016/10/05 2:12, Robert Haas wrote:

> Attached revised patches.

Few assorted review comments for 0001-Catalog*:


1.
@@ -1775,6 +1775,12 @@ BeginCopyTo(ParseState *pstate,
{
..
+ else if (rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
+ ereport(ERROR,
+ (errcode(ERRCODE_WRONG_OBJECT_TYPE),
+ errmsg("cannot copy from partitioned table \"%s\"",
+ RelationGetRelationName(rel)),
+ errhint("Try the COPY (SELECT ...) TO variant.")));
..
}

Why is this restriction?  Won't it be useful to allow it for the cases
when user wants to copy the data of all the partitions?


2.
+ if (!pg_strcasecmp(stmt->partspec->strategy, "list") &&
+ partnatts > 1)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_OBJECT_DEFINITION),
+ errmsg("cannot list partition using more than one column")));

/cannot list/cannot use list

3.
@@ -77,7 +77,7 @@ typedef enum DependencyType DEPENDENCY_INTERNAL = 'i', DEPENDENCY_EXTENSION = 'e',
DEPENDENCY_AUTO_EXTENSION= 'x',
 
- DEPENDENCY_PIN = 'p'
+ DEPENDENCY_PIN = 'p',} DependencyType;

Why is this change required?

4.
@@ -0,0 +1,69 @@
+/*-------------------------------------------------------------------------
+ *
+ * pg_partitioned_table.h
+ *  definition of the system "partitioned table" relation
+ *  along with the relation's initial contents.
+ *
+ *
+ * Portions Copyright (c) 1996-2015, PostgreSQL Global Development Group

Copyright year should be 2016.

5.
+/*
+ * PartitionSpec - partition key definition including the strategy
+ *
+ * 'strategy' partition strategy name ('list', 'range', etc.)

etc. in above comment seems to be unnecessary.

6.
+ {PartitionedRelationId, /* PARTEDRELID */

Here PARTEDRELID sounds inconvenient, how about PARTRELID?



-- 
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

Thanks for the review!

On 2016/10/25 20:32, Amit Kapila wrote:
> On Thu, Oct 6, 2016 at 12:44 PM, Amit Langote
> <Langote_Amit_f8@lab.ntt.co.jp> wrote:
>> On 2016/10/05 2:12, Robert Haas wrote:
> 
>> Attached revised patches.
> 
> Few assorted review comments for 0001-Catalog*:
> 
> 
> 1.
> @@ -1775,6 +1775,12 @@ BeginCopyTo(ParseState *pstate,
> {
> ..
> + else if (rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
> + ereport(ERROR,
> + (errcode(ERRCODE_WRONG_OBJECT_TYPE),
> + errmsg("cannot copy from partitioned table \"%s\"",
> + RelationGetRelationName(rel)),
> + errhint("Try the COPY (SELECT ...) TO variant.")));
> ..
> }
> 
> Why is this restriction?  Won't it be useful to allow it for the cases
> when user wants to copy the data of all the partitions?

Sure, CopyTo() can be be taught to scan leaf partitions when a partitioned
table is specified, but I thought this may be fine initially.

> 2.
> + if (!pg_strcasecmp(stmt->partspec->strategy, "list") &&
> + partnatts > 1)
> + ereport(ERROR,
> + (errcode(ERRCODE_INVALID_OBJECT_DEFINITION),
> + errmsg("cannot list partition using more than one column")));
> 
> /cannot list/cannot use list

Actually "list partition" works here as a verb, as in "to list partition".

> 3.
> @@ -77,7 +77,7 @@ typedef enum DependencyType
>   DEPENDENCY_INTERNAL = 'i',
>   DEPENDENCY_EXTENSION = 'e',
>   DEPENDENCY_AUTO_EXTENSION = 'x',
> - DEPENDENCY_PIN = 'p'
> + DEPENDENCY_PIN = 'p',
>  } DependencyType;
> Why is this change required?

Looks like a leftover hunk from previous revision.  Will fix.

> 4.
> @@ -0,0 +1,69 @@
> +/*-------------------------------------------------------------------------
> + *
> + * pg_partitioned_table.h
> + *  definition of the system "partitioned table" relation
> + *  along with the relation's initial contents.
> + *
> + *
> + * Portions Copyright (c) 1996-2015, PostgreSQL Global Development Group
> 
> Copyright year should be 2016.

Oops, will fix.

> 5.
> +/*
> + * PartitionSpec - partition key definition including the strategy
> + *
> + * 'strategy' partition strategy name ('list', 'range', etc.)
> 
> etc. in above comment seems to be unnecessary.

Will fix, although  I thought that list is yet incomplete.

> 6.
> + {PartitionedRelationId, /* PARTEDRELID */
> 
> Here PARTEDRELID sounds inconvenient, how about PARTRELID?

Agreed.  There used to be another catalog which had used up PARTRELID, but
that's no longer an issue.

I will include these changes in the next version of patches I will post
soon in reply to [1].

Thanks,
Amit

[1]
https://www.postgresql.org/message-id/CA%2BTgmoYJcUTcN7vVgg54GHtffH11JJWYZnfF4KiRxjV-iaACQg%40mail.gmail.com

On 2016/10/25 15:58, Amit Kapila wrote:
> On Thu, Oct 6, 2016 at 12:44 PM, Amit Langote
> <Langote_Amit_f8@lab.ntt.co.jp> wrote:
>> On 2016/10/05 2:12, Robert Haas wrote:
>>> Hmm, do we ever fire triggers on the parent for operations on a child
>>> table?  Note this thread, which seems possibly relevant:
>>>
>>> https://www.postgresql.org/message-id/flat/cd282adde5b70b20c57f53bb9ab75e27%40biglumber.com
>>
>> The answer to your question is no.
>>
>> The thread you quoted discusses statement-level triggers and the
>> conclusion is that they don't work as desired for UPDATE and DELETE on
>> inheritance tables.  As things stand, only UPDATE or DELETE on the parent
>> affects the child tables and it's proposed there that the statement-level
>> triggers on the parent and also on any child tables affected should be
>> fired in that case.
>>
> 
> Doesn't that imply that the statement level triggers should be fired
> for all the tables that get changed for statement?  If so, then in
> your case it should never fire for parent table, which means we could
> disallow statement level triggers as well on parent tables?

I may have misunderstood statement-level triggers, but don't they apply to
tables *specified* as the target table in the statement, instead of those
*changed* by resulting actions?

Now in case of inheritance, unless ONLY is specified, all tables in the
hierarchy including the parent are *implicitly* specified to be affected
by an UPDATE or DELETE operation.  So, if some or all of those tables have
any statement-level triggers defined, they should get fired.  That was the
conclusion of that thread, but that TODO item still remains [1].

I am not (or no longer) sure how that argument affects INSERT on
partitioned tables with tuple-routing though.  Are partitions at all
levels *implicitly specified to be affected* when we say INSERT INTO
root_partitioned_table?

> Some of the other things that we might want to consider disallowing on
> parent table could be:
> a. Policy on table_name

Perhaps.  Since there are no rows in the parent table(s) itself of a
partition hierarchy, it might not make sense to continue to allow creating
row-level security policies on them.

> b. Alter table has many clauses, are all of those allowed and will it
> make sense to allow them?

Currently, we only disallow the following with partitioned parent tables
as far as alter table is concerned.

- cannot change inheritance by ALTER TABLE partitioned_table INHERIT ...

- cannot let them be regular inheritance parents either - that is, the following is disallowed: ALTER TABLE some_able
INHERITpartitioned_table
 

- cannot create UNIQUE, PRIMARY KEY, FOREIGN KEY, EXCLUDE constraints

- cannot drop column involved in the partitioning key

Most other forms that affect attributes and constraints follow the regular
inheritance behavior (recursion) with certain exceptions such as:

- cannot add/drop an attribute or check constraint to *only* to/from the parent

- cannot add/drop NOT NULL constraint to/from *only* the parent

Thoughts?

Thanks,
Amit

On Wed, Oct 26, 2016 at 6:36 AM, Amit Langote
<Langote_Amit_f8@lab.ntt.co.jp> wrote:
>> 1.
>> @@ -1775,6 +1775,12 @@ BeginCopyTo(ParseState *pstate,
>> {
>> ..
>> + else if (rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
>> + ereport(ERROR,
>> + (errcode(ERRCODE_WRONG_OBJECT_TYPE),
>> + errmsg("cannot copy from partitioned table \"%s\"",
>> + RelationGetRelationName(rel)),
>> + errhint("Try the COPY (SELECT ...) TO variant.")));
>> ..
>> }
>>
>> Why is this restriction?  Won't it be useful to allow it for the cases
>> when user wants to copy the data of all the partitions?
>
> Sure, CopyTo() can be be taught to scan leaf partitions when a partitioned
> table is specified, but I thought this may be fine initially.
>

Okay, I don't want to add anything to your existing work unless it is
important.  However, I think there should be some agreement on which
of the restrictions are okay for first version of patch.  This can
avoid such questions in future from other reviewers.


>> 2.
>> + if (!pg_strcasecmp(stmt->partspec->strategy, "list") &&
>> + partnatts > 1)
>> + ereport(ERROR,
>> + (errcode(ERRCODE_INVALID_OBJECT_DEFINITION),
>> + errmsg("cannot list partition using more than one column")));
>>
>> /cannot list/cannot use list
>
> Actually "list partition" works here as a verb, as in "to list partition".
>

I am not an expert of this matter, so probably some one having better
grip can comment.  Are we using something similar in any other error
message?


-- 
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

On 2016/10/26 11:41, Amit Kapila wrote:
> On Wed, Oct 26, 2016 at 6:36 AM, Amit Langote
> <Langote_Amit_f8@lab.ntt.co.jp> wrote:
>>> 1.
>>> @@ -1775,6 +1775,12 @@ BeginCopyTo(ParseState *pstate,
>>> {
>>> ..
>>> + else if (rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
>>> + ereport(ERROR,
>>> + (errcode(ERRCODE_WRONG_OBJECT_TYPE),
>>> + errmsg("cannot copy from partitioned table \"%s\"",
>>> + RelationGetRelationName(rel)),
>>> + errhint("Try the COPY (SELECT ...) TO variant.")));
>>> ..
>>> }
>>>
>>> Why is this restriction?  Won't it be useful to allow it for the cases
>>> when user wants to copy the data of all the partitions?
>>
>> Sure, CopyTo() can be be taught to scan leaf partitions when a partitioned
>> table is specified, but I thought this may be fine initially.
>>
> 
> Okay, I don't want to add anything to your existing work unless it is
> important.  However, I think there should be some agreement on which
> of the restrictions are okay for first version of patch.  This can
> avoid such questions in future from other reviewers.

OK, so I assume you don't find this particular restriction problematic in
long term.

>>> 2.
>>> + if (!pg_strcasecmp(stmt->partspec->strategy, "list") &&
>>> + partnatts > 1)
>>> + ereport(ERROR,
>>> + (errcode(ERRCODE_INVALID_OBJECT_DEFINITION),
>>> + errmsg("cannot list partition using more than one column")));
>>>
>>> /cannot list/cannot use list
>>
>> Actually "list partition" works here as a verb, as in "to list partition".
>>
> 
> I am not an expert of this matter, so probably some one having better
> grip can comment.  Are we using something similar in any other error
> message?

In fact, I changed to the current text after Robert suggested the same [1].

Thanks,
Amit

[1]
https://www.postgresql.org/message-id/CA%2BTgmoaPxXJ14eDVia514UiuQAXyZGqfbz8Qg3G4a8Rz2gKF7w%40mail.gmail.com

On Wed, Oct 26, 2016 at 8:27 AM, Amit Langote
<Langote_Amit_f8@lab.ntt.co.jp> wrote:
> On 2016/10/26 11:41, Amit Kapila wrote:
>> On Wed, Oct 26, 2016 at 6:36 AM, Amit Langote
>> <Langote_Amit_f8@lab.ntt.co.jp> wrote:
>>>> 1.
>>>> @@ -1775,6 +1775,12 @@ BeginCopyTo(ParseState *pstate,
>>>> {
>>>> ..
>>>> + else if (rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
>>>> + ereport(ERROR,
>>>> + (errcode(ERRCODE_WRONG_OBJECT_TYPE),
>>>> + errmsg("cannot copy from partitioned table \"%s\"",
>>>> + RelationGetRelationName(rel)),
>>>> + errhint("Try the COPY (SELECT ...) TO variant.")));
>>>> ..
>>>> }
>>>>
>>>> Why is this restriction?  Won't it be useful to allow it for the cases
>>>> when user wants to copy the data of all the partitions?
>>>
>>> Sure, CopyTo() can be be taught to scan leaf partitions when a partitioned
>>> table is specified, but I thought this may be fine initially.
>>>
>>
>> Okay, I don't want to add anything to your existing work unless it is
>> important.  However, I think there should be some agreement on which
>> of the restrictions are okay for first version of patch.  This can
>> avoid such questions in future from other reviewers.
>
> OK, so I assume you don't find this particular restriction problematic in
> long term.
>

I think you can keep it as you have in patch.  After posting your
updated patches, please do send a list of restrictions which this
patch is imposing based on the argument that for first version they
are not essential.

-- 
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

On 2016/10/26 12:09, Amit Kapila wrote:
> On Wed, Oct 26, 2016 at 8:27 AM, Amit Langote
> <Langote_Amit_f8@lab.ntt.co.jp> wrote:
>> On 2016/10/26 11:41, Amit Kapila wrote:
>>> On Wed, Oct 26, 2016 at 6:36 AM, Amit Langote
>>> <Langote_Amit_f8@lab.ntt.co.jp> wrote:
>>>>
>>>> Sure, CopyTo() can be be taught to scan leaf partitions when a partitioned
>>>> table is specified, but I thought this may be fine initially.
>>>
>>> Okay, I don't want to add anything to your existing work unless it is
>>> important.  However, I think there should be some agreement on which
>>> of the restrictions are okay for first version of patch.  This can
>>> avoid such questions in future from other reviewers.
>>
>> OK, so I assume you don't find this particular restriction problematic in
>> long term.
> 
> I think you can keep it as you have in patch.  After posting your
> updated patches, please do send a list of restrictions which this
> patch is imposing based on the argument that for first version they
> are not essential.

OK, agreed that it will be better to have all such restrictions and
limitations of the first version listed in one place, rather than being
scattered across different emails where they might have been mentioned and
discussed.

I will try to include such a list when posting the latest set of patches.

Thanks,
Amit

On Wed, Oct 5, 2016 at 7:20 AM, Robert Haas <robertmhaas@gmail.com> wrote:
> On Tue, Oct 4, 2016 at 4:02 AM, Amit Langote
> <Langote_Amit_f8@lab.ntt.co.jp> wrote:
>> [ latest patch set ]
>
> Reviewing 0003:
>
>
> 5. I wonder how well this handles multi-column partition keys.  You've
> just got one Datum flag and one is-finite flag per partition, but I
> wonder if you don't need to track is-finite on a per-column basis, so
> that you could partition on (a, b) and have the first partition go up
> to (10, 10), the second to (10, infinity), the third to (20, 10), the
> fourth to (20, infinity), and the last to (infinity, infinity).  FWIW,
> Oracle supports this sort of thing, so perhaps we should, too.  On a
> related note, I'm not sure it's going to work to treat a composite
> partition key as a record type.  The user may want to specify a
> separate opfamily and collation for each column, not just inherit
> whatever the record behavior is.  I'm not sure if that's what you are
> doing, but the relcache structures don't seem adapted to storing one
> Datum per partitioning column per partition, but rather just one Datum
> per partition, and I'm not sure that's going to work very well.
>

@@ -123,6 +123,9 @@ typedef struct RelationData
{
.. MemoryContext rd_partkeycxt; /* private memory cxt for the below */ struct PartitionKeyData *rd_partkey; /*
partitionkey, or NULL */
 
+ MemoryContext rd_pdcxt; /* private context for partdesc */
+ struct PartitionDescData *rd_partdesc; /* partitions, or NULL */
+ List   *rd_partcheck; /* partition CHECK quals */
..
}

I think one thing to consider here is the increase in size of relcache
due to PartitionDescData.  I think it will be quite useful in some of
the cases like tuple routing.  Isn't it feasible to get it in some
other way, may be by using relpartbound from pg_class tuple?


-- 
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

On 2016/10/26 17:57, Amit Kapila wrote:
> @@ -123,6 +123,9 @@ typedef struct RelationData
> {
> ..
>   MemoryContext rd_partkeycxt; /* private memory cxt for the below */
>   struct PartitionKeyData *rd_partkey; /* partition key, or NULL */
> + MemoryContext rd_pdcxt; /* private context for partdesc */
> + struct PartitionDescData *rd_partdesc; /* partitions, or NULL */
> + List   *rd_partcheck; /* partition CHECK quals */
> ..
> }
> 
> I think one thing to consider here is the increase in size of relcache
> due to PartitionDescData.  I think it will be quite useful in some of
> the cases like tuple routing.  Isn't it feasible to get it in some
> other way, may be by using relpartbound from pg_class tuple?

Whereas pg_class.relpartbound stores partition bound of the *individual
partitions* in Node form, the above relcache struct is associated with
parent tables; it contains some efficient to use (and fairly compact)
representation of bounds of *all* the partitions of the parent.  Consider
for example, an array of sorted range bounds for range partitioned tables.

Thanks,
Amit

On Wed, Oct 26, 2016 at 3:04 PM, Amit Langote
<Langote_Amit_f8@lab.ntt.co.jp> wrote:
> On 2016/10/26 17:57, Amit Kapila wrote:
>> @@ -123,6 +123,9 @@ typedef struct RelationData
>> {
>> ..
>>   MemoryContext rd_partkeycxt; /* private memory cxt for the below */
>>   struct PartitionKeyData *rd_partkey; /* partition key, or NULL */
>> + MemoryContext rd_pdcxt; /* private context for partdesc */
>> + struct PartitionDescData *rd_partdesc; /* partitions, or NULL */
>> + List   *rd_partcheck; /* partition CHECK quals */
>> ..
>> }
>>
>> I think one thing to consider here is the increase in size of relcache
>> due to PartitionDescData.  I think it will be quite useful in some of
>> the cases like tuple routing.  Isn't it feasible to get it in some
>> other way, may be by using relpartbound from pg_class tuple?
>
> Whereas pg_class.relpartbound stores partition bound of the *individual
> partitions* in Node form, the above relcache struct is associated with
> parent tables; it contains some efficient to use (and fairly compact)
> representation of bounds of *all* the partitions of the parent.
>

Okay, but still it will be proportional to number of partitions and
the partition keys.  Is it feasible to store ranges only for
partitions that are actively accessed?  For example, consider a table
with 100 partitions and the first access to table requires to access
5th partition, then we store ranges for first five partitions or
something like that.  This could be helpful, if we consider cases that
active partitions are much less as compare to total partitions of a
table.

-- 
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

On Wed, Oct 26, 2016 at 6:12 AM, Amit Kapila <amit.kapila16@gmail.com> wrote:
> On Wed, Oct 26, 2016 at 3:04 PM, Amit Langote
> <Langote_Amit_f8@lab.ntt.co.jp> wrote:
>> On 2016/10/26 17:57, Amit Kapila wrote:
>>> @@ -123,6 +123,9 @@ typedef struct RelationData
>>> {
>>> ..
>>>   MemoryContext rd_partkeycxt; /* private memory cxt for the below */
>>>   struct PartitionKeyData *rd_partkey; /* partition key, or NULL */
>>> + MemoryContext rd_pdcxt; /* private context for partdesc */
>>> + struct PartitionDescData *rd_partdesc; /* partitions, or NULL */
>>> + List   *rd_partcheck; /* partition CHECK quals */
>>> ..
>>> }
>>>
>>> I think one thing to consider here is the increase in size of relcache
>>> due to PartitionDescData.  I think it will be quite useful in some of
>>> the cases like tuple routing.  Isn't it feasible to get it in some
>>> other way, may be by using relpartbound from pg_class tuple?
>>
>> Whereas pg_class.relpartbound stores partition bound of the *individual
>> partitions* in Node form, the above relcache struct is associated with
>> parent tables; it contains some efficient to use (and fairly compact)
>> representation of bounds of *all* the partitions of the parent.
>>
>
> Okay, but still it will be proportional to number of partitions and
> the partition keys.  Is it feasible to store ranges only for
> partitions that are actively accessed?  For example, consider a table
> with 100 partitions and the first access to table requires to access
> 5th partition, then we store ranges for first five partitions or
> something like that.  This could be helpful, if we consider cases that
> active partitions are much less as compare to total partitions of a
> table.

I have serious doubt about whether it's a good idea to do that EVER,
but it certainly doesn't need to be in the first version of this
patch.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Tue, Oct 25, 2016 at 2:58 AM, Amit Kapila <amit.kapila16@gmail.com> wrote:
> a. Policy on table_name

No, because queries against the parent will apply the policy to
children.  See today's commit
162477a63d3c0fd1c31197717140a88077a8d0aa.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Tue, Oct 25, 2016 at 9:06 PM, Amit Langote
<Langote_Amit_f8@lab.ntt.co.jp> wrote:
> I will include these changes in the next version of patches I will post
> soon in reply to [1].
> [1]
> https://www.postgresql.org/message-id/CA%2BTgmoYJcUTcN7vVgg54GHtffH11JJWYZnfF4KiRxjV-iaACQg%40mail.gmail.com

How soon?  Tempus fugit, and tempus in this release cycle fugit
particularly quickly.  It looks like the last revision of these
patches was on October 7th, and that's now more than 2 weeks ago.  We
need to get moving here if this is going to happen this cycle.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Tue, Oct 25, 2016 at 10:00 PM, Amit Langote
<Langote_Amit_f8@lab.ntt.co.jp> wrote:
> I am not (or no longer) sure how that argument affects INSERT on
> partitioned tables with tuple-routing though.  Are partitions at all
> levels *implicitly specified to be affected* when we say INSERT INTO
> root_partitioned_table?

I'd say yes.

>> Some of the other things that we might want to consider disallowing on
>> parent table could be:
>> a. Policy on table_name
>
> Perhaps.  Since there are no rows in the parent table(s) itself of a
> partition hierarchy, it might not make sense to continue to allow creating
> row-level security policies on them.

No, per my previous email.  Those policies are emphatically not without effect.

>> b. Alter table has many clauses, are all of those allowed and will it
>> make sense to allow them?
>
> Currently, we only disallow the following with partitioned parent tables
> as far as alter table is concerned.
>
> - cannot change inheritance by ALTER TABLE partitioned_table INHERIT ...
>
> - cannot let them be regular inheritance parents either - that is, the
>   following is disallowed: ALTER TABLE some_able INHERIT partitioned_table
>
> - cannot create UNIQUE, PRIMARY KEY, FOREIGN KEY, EXCLUDE constraints
>
> - cannot drop column involved in the partitioning key
>
> Most other forms that affect attributes and constraints follow the regular
> inheritance behavior (recursion) with certain exceptions such as:
>
> - cannot add/drop an attribute or check constraint to *only* to/from
>   the parent
>
> - cannot add/drop NOT NULL constraint to/from *only* the parent
>
> Thoughts?

Seems sensible to me.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 2016/10/27 3:13, Robert Haas wrote:
> On Tue, Oct 25, 2016 at 9:06 PM, Amit Langote
> <Langote_Amit_f8@lab.ntt.co.jp> wrote:
>> I will include these changes in the next version of patches I will post
>> soon in reply to [1].
>> [1]
>> https://www.postgresql.org/message-id/CA%2BTgmoYJcUTcN7vVgg54GHtffH11JJWYZnfF4KiRxjV-iaACQg%40mail.gmail.com
> 
> How soon?  Tempus fugit, and tempus in this release cycle fugit
> particularly quickly.  It looks like the last revision of these
> patches was on October 7th, and that's now more than 2 weeks ago.  We
> need to get moving here if this is going to happen this cycle.

Sorry about the delay, I'll post no later than tomorrow.

Thanks,
Amit

On 2016/10/05 10:50, Robert Haas wrote:
> On Tue, Oct 4, 2016 at 4:02 AM, Amit Langote
> <Langote_Amit_f8@lab.ntt.co.jp> wrote:
>> [ latest patch set ]
>
> Reviewing 0003:

Thanks a lot for the review and sorry about the delay in replying.

> +      This form attaches an existing table (partitioned or otherwise) as
>
> (which might itself be partitioned)
>
> +      partition of the target table.  Partition bound specification must
> +      correspond with the partition method and the key of the target table.
>
> The partition bound specification must correspond to the partitioning
> method and partitioning key of the target table.
>
> +      The table being attached must have all the columns of the target table
> +      with matching types and no more. Also, it must have all the matching
>
> The table to be attached must have all of the same columns as the
> target table and no more; moreover, the column types must also match.

Done.

> +      with matching types and no more. Also, it must have all the matching
> +      constraints as the target table.  That includes both <literal>NOT NULL</>
> +      and <literal>CHECK</> constraints.  If some <literal>CHECK</literal>
> +      constraint of the table being attached is marked <literal>NO
> INHERIT</literal>,
> +      the command will fail; such constraints must either be dropped or
> +      recreated without the <literal>NO INHERIT</literal> mark.
>
> Why all of these requirements?

Except the last one, regular inheritance currently has the same
requirements; from the docs:

"""
To be added as a child, the target table must already contain all the same
columns as the parent (it could have additional columns, too). The columns
must have matching data types, and if they have NOT NULL constraints in
the parent then they must also have NOT NULL constraints in the child.

There must also be matching child-table constraints for all CHECK
constraints of the parent, except those marked non-inheritable (that is,
created with ALTER TABLE ... ADD CONSTRAINT ... NO INHERIT) in the parent,
which are ignored; all child-table constraints matched must not be marked
non-inheritable.
"""

Note that the "(it could have additional columns, too)" part does not
apply for partitions.  Also the last point does not hold for partitioned
tables because they never have NO INHERIT constraints.

> We could instead perform a scan to
> validate that the constraints are met.  I think the way this should
> work is:
>
> 1. ATTACH PARTITION works whether matching NOT NULL and CHECK
> constraints are present or not.

Wouldn't this break the invariant that if the parent has a NOT NULL or a
CHECK constraint, all of its children must have it too?  Remember that we
allow directly inserting/updating data into partitions, so these
constraints better be persistent.

> 2. If all of the constraints are present, and a validated constraint
> matching the implicit partitioning constraint is also present, then
> ATTACH PARTITION does not scan the table to validate constraints;
> otherwise, it does.

I wonder what matching in "validated constraint matching with the implicit
partitioning constraint" means?  By that do you mean an exact equal()
match, provided both have been suitably canonicalized?  Or an existing
check constraint that *implies* the partitioning constraint?

> 3. NO VALIDATE is not an option.

I had my doubts about this option initially, but 2 above seemed hard to
do.  Or perhaps we can just remove this option and we will always scan
when attaching a partition.

> +      Currently <literal>UNIQUE</literal>, <literal>PRIMARY KEY</literal>, and
> +      <literal>FOREIGN KEY</literal> constraints are not considered, but that
> +      might change in the future.
>
> Really?  Changing that sounds impractical to me.

Assuming you are questioning the "but that might change in the future"
part, I took that line from the description for INHERIT.  Maybe it was
written in that section with a view that someone might implement globally
enforced variant of each of those constraints.  I thought the path to that
would be slightly easier with declarative partitioning but that may be too
optimistic.  Removed that part.

> +      This form detaches specified partition of the target table.  The
> +      detached partition continues to exist as a standalone table with no ties
> +      remaining with the target table.
>
> continues to exist as a standalone table, but no longer has any ties
> to the table from which it was detached.

Done.

>
> +      Note that if a partition being detached is itself a partitioned table,
> +      it continues to exist as such.
>
> You don't really need to say this, I think.  All of the properties of
> the detached table are retained, not only its partitioning status.
> You wouldn't like it if I told you to document "note that if a
> partition being detached is unlogged, it will still be unlogged".

I thought same as Petr who replied to this part of your email - might help
avoid confusion.  But it might be redundant, so I removed that line.

>
>     To add the table as a new child of a parent table, you must own the
> -   parent table as well.
> +   parent table as well.  That applies to both adding the table as a
> +   inheritance child of a parent table and attaching a table as partition to
> +   the table.
>
> To add the table as a new child of a parent table, or as a new
> partition of a partitioned table, you must own the parent table as
> well.

Done.

> +        The name of the table to attach as a new partition to or
> detach from this table.
>
> s/to or/or to/

Oops, fixed.

> +    <literal>NO VALIDATE</> option is spcified.
>
> Typo, but see comments above about nuking this altogether.

Fixed the typo for now. As I said above it's not clear to me how the
alternative method of skipping the scan is supposed to work.

>      A recursive <literal>DROP COLUMN</literal> operation will remove a
>      descendant table's column only if the descendant does not inherit
>      that column from any other parents and never had an independent
> -    definition of the column.  A nonrecursive <literal>DROP
> +    definition of the column (which always holds if the descendant table
> +    is a partition).  A nonrecursive <literal>DROP
>      COLUMN</literal> (i.e., <command>ALTER TABLE ONLY ... DROP
>      COLUMN</command>) never removes any descendant columns, but
> -    instead marks them as independently defined rather than inherited.
> +    instead marks them as independently defined rather than inherited,
> +    unless the descendant table is a partition.
>
> This is a hairy explanation.  I suggest that the documentation say
> this (and that the code implement it): A nonrecursive DROP TABLE
> command will fail for a partitioned table, because all partitions of a
> table must have the same columns as the partitioning root.

Agreed, done (including the code).

> -    that are not marked <literal>NO INHERIT</>.
> +    that are not marked <literal>NO INHERIT</> which are unsupported if
> +    the table is a partitioned table.
>
> I think you can omit this hunk.

Done.

>
> +   If <literal>PARTITION OF</literal> clause is specified then the table is
> +   created as a partition of <literal>parent_table</literal> with specified
> +   bounds.  However, unlike regular tables, one cannot specify
> +   <literal>PARTITION BY</literal> clause which means foreign tables can
> +   only be created as leaf partitions.
>
> I'd delete the sentence beginning with "However".

Done.

> +   Create foreign table <structname>measurement_y2016m07</>, which will be
> +   accessed through the server <structname>server_07</>, that is partition
> +   of the range partitioned table <structname>measurement</>:
>
> s/, that is/ as a/

Fixed.

> +<phrase>and <replaceable
> class="PARAMETER">partition_bound_spec</replaceable> is:</phrase>
> +
> +FOR VALUES { <replaceable class="PARAMETER">list_spec</replaceable> |
> <replaceable class="PARAMETER">range_spec</replaceable> }
>
> I think you can inline the definitions of list_spec and range_spec
> here instead of making them separate productions, and I think that
> would be preferable.
>
> FOR VALUES { IN ( <replaceable
> class="PARAMETER">expression</replaceable> [, ...] ) |
> START <replaceable class="PARAMETER">lower-bound</replaceable> [
> INCLUSIVE | EXCLUSIVE ] END <replaceable
> class="PARAMETER">upper-bound</replaceable> [ INCLUSIVE | EXCLUSIVE ]
> }

Ah, done that way.

>
> +      parent table (name optionally schema-qualified).
>
> Parenthetical phrase isn't needed.

Removed.

>
> +      A partition bound specification must be present and must correspond with
> +      partition method and key of the parent table.  It is checked using the
> +      specification that the new partition does not overlap with any existing
> +      partitions of the parent.
>
> The partition bound specification must correspond to the partitioning
> method and partitioning key of the parent table, and must not overlap
> with any existing partition of that parent.

Done.

> +      clause, if any.  Defaults and constraints can optionally be specified
> +      for each of the inherited columns, which override those in the parent.
>
> Surely not.  You can't "override" an inherited constraint or an
> inherited default.  The child may have extra constraints not present
> in the parent, and may have different defaults when it is directly
> targeted by an insertion, but it can't possibly override the parent
> defaults.

Removed ", which override those in the parent.".

>
> +      One can also specify table constraints, in addition to those inherited
> +      from the parent.  Note that all subsequent schema modifications to the
> +      parent propagate to partition.
>
> The first part of this seems right, but then what's going on with the
> reference to constraints in the previous sentence?  (See previous
> review comment.) The second sentence I would delete (but see below).

In the previous sentence, I meant the constraints that one can specify
using WITH OPTIONS [ column_constraint [...] ], but I removed
the "overrides those in the parent" part as mentioned.

Removed the second sentence and instead adopted your text below.

>
> +     <para>
> +      Any data row subsequently inserted into the parent table is mapped to
> +      and stored in the partition, provided partition key of the row falls
> +      within the partition bounds.
> +     </para>
>
> How about: Rows inserted into a partitioned table will be
> automatically routed to the correct partition.  If no suitable
> partition exists, an error will occur.

Much better, done.

> +     <para>
> +      A partition is dropped or truncated when the parent table is dropped or
> +      truncated.  Dropping it directly using <literal>DROP TABLE</literal>
> +      will fail; it must first be <firstterm>detached</> from the parent.
> +      However, truncating a partition directly works.
> +     </para>
>
> How about: A partition must have the same column names and types as
> the table of which it is a partition.  Therefore, modifications the
> column names or types of the partitioned table will automatically
> propagate to all children, as will operations such as TRUNCATE which
> normally affect a table and all of its inheritance children.  It is
> also possible to TRUNCATE a partition individually, just as for an
> inheritance child.

Done.

> Insisting that you can't drop a child without detaching it first seems
> wrong to me.  If I already made this comment and you responded to it,
> please point me back to whatever you said.  However, my feeling is
> this is flat wrong and absolutely must be changed.

I said the following [1]:

| Hmm, I don't think I like this.  Why should it be necessary to detach
| a partition before dropping it?  That seems like an unnecessary step.

I thought we had better lock the parent table when removing one of its
partitions and it seemed a bit odd to lock the parent table when dropping
a partition using DROP TABLE?  OTOH, with ALTER TABLE parent DETACH
PARTITION, the parent table is locked anyway.

> -            if (is_no_inherit)
> +
> +            /* Discard the NO INHERIT flag if the relation is a partition */
> +            if (is_no_inherit && !rel->rd_rel->relispartition)
>
> Something about this seems fishy.  In StoreRelCheck(), you disallow
> the addition of a NO INHERIT constraint on a partition, but here you
> seem happy to accept it and ignore the NO INHERIT property.  That
> doesn't seem consistent.  We should adopt a consistent policy about
> what to do about such constraints, and I submit that throwing an error
> is better than silently changing things, unless you have some reason
> for doing that which I'm not seeing.  Anyway, we should have the same
> behavior in both cases.

Allowing to add NO INHERIT constraints to (leaf) partitions does not seem
like it will cause any problems not that the flag will have any meaning.
It will come into play only in one case - in the form of causing an error
if a constraint with the same name is added to the parent.

With that in mind, I removed the check in StoreRelCheck() that you mention
here and also removed the above hunk to which your comment was addressed.

> We should also standardize on what value of conislocal and coninhcount
> children end up with; presumably the latter should be 1, but I'm not
> sure if the former should be true or false.  In either case, anything
> that can vary between children probably needs to be dumped, so let's
> enforce that it doesn't so we don't have to dump it.  I'm not sure
> whether the code here achieves those objectives, though, and note the
> comment in the function header about making sure the logic here
> matches MergeConstraintsIntoExisting.

To summarize:

If a table is partition, it doesn't have any local attributes. Also, it
doesn't have any check constraint that is both local *and* inherited.
Only the non-inherited constraints are ever local.

So, all parent attributes will be present present in all the partitions at
any given time with attislocal = false and attinhcount = 1.  Likewise for
inherited constraints with conislocal = false and coninhcount = 1.

With above rules in place, pg_dump no longer dumps attributes or inherited
constraints in CREATE TABLE commands of individual partitions.

> I think the overriding principle here should be: If you attach a table
> as a partition, it must not be part of a standard inheritance
> hierarchy, and it must not be a partition of any other table.  It can,
> however, be partitioned itself.  If you later detach a partition, it
> ends up as a standalone table with a copy of each constraint it had as
> a partition - probably including the implicit partition constraint.
> The DBA can drop those constraints if they're not wanted.

Check.  About implicit constraints, see below.

> I wonder if it's really a good idea for the partition constraints to
> be implicit; what is the benefit of leaving those uncatalogued?

I did start out that way - ie, catalogued implicit constraints, but later
thought it might not be good to end up with multiple copies of essentially
the same information.  With cataloguing them will come dependencies and
all places that know about pg_constraint.

In the long term, I think we're only going to need them because we want to
enforce them when directly inserting data into partitions.

So, detaching a partition does not emit a check constraint matching the
implicit partition constraint.

> + * Depending on whether the relation in question is list or range
> + * partitioned, one of the fields is set.
> + */
> +typedef struct BoundCollectionData
> +{
> +    struct ListInfo       *listinfo;
> +    struct RangeInfo   *rangeinfo;
> +} BoundCollectionData;
>
> This seems like an odd design.  First, when you have a pointer to
> either of two things, the normal tool for that in C would be a union,
> not a struct. Second, in PostgreSQL we typically handle that by making
> both of the things nodes and then you can use IsA() or switch on
> nodeTag() to figure out what you've got.  Third, the only place this
> is used at least in 0003 is as part of PartitionDescData, which only
> has 3 members, so if you were going to do it with 2 members, you could
> just include these two members directly.  Considering all of the
> foregoing, I'd suggest making this a union and including partstrategy
> in PartitionDescData.
>
> I think that the names ListInfo and RangeInfo are far too generic for
> something that's specific to partitioning.

How about this:

/*
 * Collection of bounds of a partitioned relation (either physical or
 * logical)
 */
typedef struct BoundCollectionData
{
    char    strategy;   /* list or range bounds? */

    union
    {
        struct PartitionListInfo    lists;
        struct PartitionRangeInfo   ranges;
    } bounds;
} BoundCollectionData;

I added the strategy field here instead of PartitionDescData, because this
structure is supposed to represent even the transient partitioned
relations.  If the relation is a physical relation we can determine
strategy from PartitionKey.

I renamed ListInfo/RangeInfo to PartitionListInfo/PartitionRangeInfo.

> +/*
> + * Range bound collection - sorted array of ranges of partitions of a range
> + * partitioned table
> + */
> +typedef struct RangeInfo
> +{
> +    struct PartitionRange    **ranges;
> +} RangeInfo;
> +
> +/* One partition's range */
> +typedef struct PartitionRange
> +{
> +    struct PartitionRangeBound    *lower;
> +    struct PartitionRangeBound    *upper;
> +} PartitionRange;
>
> This representation doesn't seem optimal, because in most cases the
> lower bound of one partition will be the upper bound of the next.  I
> suggest that you flatten this representation into a single list of
> relevant bounds, each flagged as to whether it is exclusive and
> whether it is finite; and a list with one more element of bounds.  For
> example, suppose the partition bounds are [10, 20), [20, 30), (30,
> 40), and [50, 60).  You first construct a list of all of the distinct
> bounds, flipping inclusive/exclusive for the lefthand bound in each
> case.  So you get:
>
> 10 EXCLUSIVE
> 20 EXCLUSIVE
> 30 EXCLUSIVE
> 30 INCLUSIVE
> 40 EXCLUSIVE
> 50 EXCLUSIVE
> 60 EXCLUSIVE
>
> When ordering items for this list, if the same item appears twice, the
> EXCLUSIVE copy always appears before INCLUSIVE.  When comparing
> against an EXCLUSIVE item, we move to the first half of the array if
> we are searching for a value strictly less than that value; when
> comparing against an INCLUSIVE item, we move to the first half of the
> array if we are searching for a value less than or equal to that
> value.
>
> This is a list of seven items, so a binary search will return a
> position between 0 (less than all items) and 7 (greater than all
> items).  So we need a list of 8 partition mappings, which in this case
> will look like this: -1, 0, 1, -1, 2, -1, 3, -1.
>
> In this particular example, there are only two adjacent partitions, so
> we end up with 7 bounds with this representation vs. 8 with yours, but
> in general I think the gains will be bigger.  If you've got 1000
> partitions and they're all adjacent, which is likely, you store 1000
> bounds instead of 2000 bounds by doing it this way.

Thanks for the idea.  I have implemented the same.

>
> + * Note: This function should be called only when it is known that 'relid'
> + * is a partition.
>
> Why?  How about "Because this function assumes that the relation whose
> OID is passed as an argument will have precisely one parent, it should
> only been called when it is known that the relation is a partition."

Rewrote the note.

> +    /*
> +     * Translate vars in the generated expression to have correct attnos.
> +     * Note that the vars in my_qual bear attnos dictated by key which carries
> +     * physical attnos of the parent.  We must allow for a case where physical
> +     * attnos of a partition can be different from the parent.
> +     */
> +    partexprs_item = list_head(key->partexprs);
> +    for (i = 0; i < key->partnatts; i++)
> +    {
> +        AttrNumber    attno = key->partattrs[i],
> +                    new_attno;
> +        char       *attname;
> +
> +        if (attno != 0)
> +        {
> +            /* Simple column reference */
> +            attname = get_attname(RelationGetRelid(parent), attno);
> +            new_attno = get_attnum(RelationGetRelid(rel), attname);
> +
> +            if (new_attno != attno)
> +                my_qual = (List *) translate_var_attno((Node *) my_qual,
> +                                                       attno,
> +                                                       new_attno);
>
> It can't really be safe to do this one attribute number at a time, or
> even if by some good fortune it can't be broken, it at least it seems
> extremely fragile.  Suppose that you translate 0 -> 3 and then 3 -> 5;
> now the result is garbage.  It's not very efficient to do this one
> attno at a time, either.

You are quite right.  I changed this to us map_variable_attnos().

When doing that, I noticed that the above function does not map whole-row
references but instead just returns whether one was encountered, so that
the caller can output an error that expressions containing whole-row vars
are not allowed in the corresponding context (examples of such callers
include transformTableLikeClause()).  In context of this function, there
is no way we could output such error.

Thinking more about that, even if we did map whole-row vars in check
expressions, constraint exclusion code would not be able to handle them,
because it does not work with anything but Const node as predicate's or
query clause's right-operand.

I went ahead and prohibited whole-row references from being used in the
partition key expressions in the first place (ie, in patch 0001).

>
> +    if (classform->relispartition)
> +        ereport(ERROR,
> +                (errcode(ERRCODE_WRONG_OBJECT_TYPE),
> +                 errmsg("\"%s\" is a partition of \"%s\"", rel->relname,
> +                        get_rel_name(get_partition_parent(relOid))),
> +                 errhint("Use ALTER TABLE DETACH PARTITION to be able
> to drop it.")));
> +
>
> RangeVarCallbackForDropRelation should do only minimal sanity checks;
> defer this until after we have a relation lock.

Moved this check to RemoveRelations(), wherein after
RangeVarGetRelidExtended() returns, the relation is guaranteed to be locked.

> I didn't get all the way through this patch, so this is a pretty
> incomplete review, but it's late here and I'm out of steam for
> tonight.  Some general comments:
>
> 1. I think that this patch seems to introduce an awful lot of new
> structures with confusingly similar names and purposes:
> PartitionBoundList, PartitionBoundRange, ListInfo, RangeInfo,
> PartitionRange, PartitionList, ListValue, RangePartition.  You've got
> 4 different structures here that all have "Partition" and "Range" in
> the name someplace, including both PartitionRange and RangePartition.
> Maybe there's no way around that kind of duplication; after all there
> are quite a few moving parts here.  But it seems like it would be good
> to try to simplify it.

OK, I got rid of most of those structs and now there are only the
following: PartitionListInfo, PartitionRangeInfo, PartitionRangeBound and
PartitionListValue.

> 2. I'm also a bit concerned about the fairly large amount of
> apparently-boilerplate code in partition.c, all having to do with how
> we create all of these data structures and translate between different
> forms of them.  I haven't understood that stuff thoroughly enough to
> have a specific idea about how we might be able to get rid of any of
> it, and maybe there's no way.  But that too seems like a topic for
> futher investigation.  One idea is that maybe some of these things
> should be nodes that piggyback on the existing infrastructure in
> src/backend/nodes instead of inventing a new way to do something
> similar.

I thought about the idea of using the src/backend/nodes infrastructure for
partitioning data structures.  However, none of those structures are part
of any existing Node nor do I imagine they will be in future, so making
them a Node seems unnecessary.  But then there is a subset of NodeTags
called "TAGS FOR RANDOM OTHER STUFF" which are Node objects that not part
of parse/plan/execute node tree structures such as T_TriggerData,
T_ForeignKeyCacheInfo.  I wonder if you meant to make partitioning
structures nodes of that category.

That said, I have tried in the current patch to reduce the amount of
unnecessary boilerplate code.

> 3. There are a lot of places where you have separate code for the
> range and list partitioning cases, and I'm suspicious that there are
> ways that code could be unified.  For example, with list partitioning,
> you have a bunch of Datums which represent the specific values that
> can appear in the various partitions, and with range partitioning, you
> have a bunch of Datums that represent the edges of partitions.  Well,
> if you used the same array for both purposes, you could use the same
> code to copy it.  That would involve flattening away a lot of the
> subsidiary structure under PartitionDescData and pulling up stuff that
> is buried lower down into the main structure, but I think that's
> likely a good idea anyway - see also point #1.

Hmm, I think it might be somewhat clearer to keep them separate in the
form of PartitionListInfo and PartitionRangeInfo structs.  In case of
range partitioning, each individual range bound is actually a
PartitionRangeBound instance, not just a Datum.  That's because we have to
store for each range bound the following information: an array of Datums
to represent the composite partition key, is-finite flags for each
partitioning column, a flag representing whether it is a inclusive bound,
and a flag representing whether it is lower or upper bound.

> 4. I'm somewhat wondering if we ought to just legislate that the lower
> bound is always inclusive and the upper bound is always exclusive.
> The decision to support both inclusive and exclusive partition bounds
> is responsible for an enormous truckload of complexity in this patch,
> and I have a feeling it is also going to be a not-infrequent cause of
> user error.

I thought we decided at some point to go with range type like notation to
specify range partition bound because of its flexibility.  I agree though
that with that flexibility, there will more input combinations that will
cause error.  As for the internal complexity, it's not clear to me whether
it will be reduced by always-inclusive lower and always-exclusive upper
bounds.  We would still need to store the inclusive flag with individual
PartitionRangeBound and consider it when comparing them with each other
and with partition key of tuples.

> 5. I wonder how well this handles multi-column partition keys.  You've
> just got one Datum flag and one is-finite flag per partition, but I
> wonder if you don't need to track is-finite on a per-column basis, so
> that you could partition on (a, b) and have the first partition go up
> to (10, 10), the second to (10, infinity), the third to (20, 10), the
> fourth to (20, infinity), and the last to (infinity, infinity).  FWIW,
> Oracle supports this sort of thing, so perhaps we should, too.  On a

I have implemented the feature that allows specifying UNBOUNDED per column.

> related note, I'm not sure it's going to work to treat a composite
> partition key as a record type.  The user may want to specify a
> separate opfamily and collation for each column, not just inherit
> whatever the record behavior is.  I'm not sure if that's what you are
> doing, but the relcache structures don't seem adapted to storing one
> Datum per partitioning column per partition, but rather just one Datum
> per partition, and I'm not sure that's going to work very well.

Actually, there *is* one Datum per partitioning column.  That is,
composite partition key is not treated as a record as it may have seemed.


Please find attached the latest version of the patches taking care of the
above review comments and some other issues I found.  As Amit Kapila
requested [2], here is a list of restrictions on partitioned tables, of
which some we might be able to overcome in future:

- cannot create indexes
- cannot create row triggers
- cannot specify UNIQUE, PRIMARY KEY, FOREIGN KEY, EXCLUDE constraints
- cannot become inheritance parent or child
- cannot use in COPY TO command

Thanks,
Amit

[1]
https://www.postgresql.org/message-id/169708f6-6e5a-18d1-707b-1b323e4a6baf%40lab.ntt.co.jp

[2]
https://www.postgresql.org/message-id/CAA4eK1LqTqZkPSoonF5_cOz94OUZG9j0PNfLdhi_nPtW82fFVA%40mail.gmail.com

On Fri, Oct 28, 2016 at 3:53 AM, Amit Langote
<Langote_Amit_f8@lab.ntt.co.jp> wrote:
>> 4. I'm somewhat wondering if we ought to just legislate that the lower
>> bound is always inclusive and the upper bound is always exclusive.
>> The decision to support both inclusive and exclusive partition bounds
>> is responsible for an enormous truckload of complexity in this patch,
>> and I have a feeling it is also going to be a not-infrequent cause of
>> user error.
>
> I thought we decided at some point to go with range type like notation to
> specify range partition bound because of its flexibility.  I agree though
> that with that flexibility, there will more input combinations that will
> cause error.  As for the internal complexity, it's not clear to me whether
> it will be reduced by always-inclusive lower and always-exclusive upper
> bounds.  We would still need to store the inclusive flag with individual
> PartitionRangeBound and consider it when comparing them with each other
> and with partition key of tuples.

We did.  But I now think that was kind of silly.  I mean, we also
talked about not having a hard distinction between list partitioning
and range partitioning, but you didn't implement it that way and I
think that's probably a good thing.  The question is - what's the
benefit of allowing this to be configurable?

For integers, there's absolutely no difference in expressive power.
If you want to allow 1000-2000 with both bounds inclusive, you can
just say START (1000) END (2001) instead of START (1000) END (2000)
INCLUSIVE.  This is also true for any other datatype where it makes
sense to talk about "the next value" and "the previous value".
Instead of making the upper bound inclusive, you can just end at the
next value instead.  If you were tempted to make the lower bound
exclusive, same thing.

For strings and numeric types that are not integers, there is in
theory a loss of power.  If you want a partition that allows very
value starting with 'a' plus the string 'b' but not anything after
that, you are out of luck.  START ('a') END ('b') INCLUSIVE would have
done exactly what you want, but now you need to store the first string
that you *don't* want to include in that partition, and what's that?
Dunno.  Or similarly if you want to store everything from 1.0 up to
and including 2.0 but nothing higher, you can't, really.

But who wants that?  People who are doing prefix-based partitioning of
their text keys are going to want all of the 'a' things together, and
all of the 'b' things in another category.  Same for ranges of
floating-point numbers, which are also probably an unlikely candidate
for a partitioning key anyway.

So let's look at the other side.  What do we gain by excluding this
functionality?  Well, we save a parser keyword: INCLUSIVE no longer
needs to be a keyword.  We also can save some code in
make_one_range_bound(), RelationBuildPartitionDesc(),
copy_range_bound(), partition_rbound_cmp(), and partition_rbound_eq().

Also, if we exclude this now as I'm proposing, we can always add it
back later if it turns out that people need it.  On the other hand, if
we include it in the first version, it's going to be very hard to get
rid of it if it turns out we don't want it.  Once we release support
for a syntax, we're kinda stuck with it.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Fri, Oct 28, 2016 at 3:53 AM, Amit Langote
<Langote_Amit_f8@lab.ntt.co.jp> wrote:
> [ new patches ]

Reviewing 0006:

This patch seems scary.  I sort of assumed from the title -- "Teach a
few places to use partition check quals." -- that this was an optional
thing, some kind of optimization from which we could reap further
advantage once the basic infrastructure was in place.  But it's not
that at all.  It's absolutely necessary that we do this, or data
integrity is fundamentally compromised.  How do we know that we've
found all of the places that need to be taught about these new,
uncatalogued constraints?

I'm feeling fairly strongly like you should rewind and make the
partitioning constraints normal catalogued constraints.  That's got a
number of advantages, most notably that we can be sure they will be
properly enforced by the entire system (modulo existing bugs, of
course).  Also, they'll show up automatically in tools like psql's \d
output, pgAdmin, and anything else that is accustomed to being able to
find constraints in the catalog.  We do need to make sure that those
constraints can't be dropped (or altered?) inappropriately, but that's
a relatively small problem.  If we stick with the design you've got
here, every client tool in the world needs to be updated, and I'm not
seeing nearly enough advantage in this system to justify that kind of
upheaval.

In fact, as far as I can see, the only advantage of this approach is
that when the insert arrives through the parent and is routed to the
child by whatever tuple-routing code we end up with (I guess that's
what 0008 does), we get to skip checking the constraint, saving CPU
cycles.  That's probably an important optimization, but I don't think
that putting the partitioning constraint in the catalog in any way
rules out the possibility of performing that optimization.  It's just
that instead of having the partitioning excluded-by-default and then
sometimes choosing to include it, you'll have it included-by-default
and then sometimes choose to exclude it.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Fri, Oct 28, 2016 at 3:53 AM, Amit Langote
<Langote_Amit_f8@lab.ntt.co.jp> wrote:
>> Insisting that you can't drop a child without detaching it first seems
>> wrong to me.  If I already made this comment and you responded to it,
>> please point me back to whatever you said.  However, my feeling is
>> this is flat wrong and absolutely must be changed.
>
> I said the following [1]:
>
> | Hmm, I don't think I like this.  Why should it be necessary to detach
> | a partition before dropping it?  That seems like an unnecessary step.
>
> I thought we had better lock the parent table when removing one of its
> partitions and it seemed a bit odd to lock the parent table when dropping
> a partition using DROP TABLE?  OTOH, with ALTER TABLE parent DETACH
> PARTITION, the parent table is locked anyway.

That "OTOH" part seems like a pretty relevant point.

Basically, I think people expect to be able to say "DROP THINGTYPE
thingname" or at most "DROP THINGTYPE thingname CASCADE" and have that
thing go away.  I'm opposed to anything which requires some other
series of steps without a very good reason, and possible surprise
about the precise locks that the command requires isn't a good enough
reason from my point of view.

>> I wonder if it's really a good idea for the partition constraints to
>> be implicit; what is the benefit of leaving those uncatalogued?
>
> I did start out that way - ie, catalogued implicit constraints, but later
> thought it might not be good to end up with multiple copies of essentially
> the same information.  With cataloguing them will come dependencies and
> all places that know about pg_constraint.
>
> In the long term, I think we're only going to need them because we want to
> enforce them when directly inserting data into partitions.

See my other email on this topic.  I agree there are some complexities
here, including making sure that pg_dump does the right thing.  But I
think it's the right way to go.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Fri, Oct 28, 2016 at 3:53 AM, Amit Langote
<Langote_Amit_f8@lab.ntt.co.jp> wrote:
>  [ new patches ]

Reviewing 0005:

Your proposed commit messages says this:

> If relation is the target table (UPDATE and DELETE), flattening is
> done regardless (scared to modify inheritance_planner() yet).

In the immortal words of Frank Herbert: “I must not fear. Fear is the
mind-killer. Fear is the little-death that brings total obliteration.
I will face my fear. I will permit it to pass over me and through me.
And when it has gone past I will turn the inner eye to see its path.
Where the fear has gone there will be nothing. Only I will remain.”

In other words, I'm not going to accept fear as a justification for
randomly-excluding the target-table case from this code.  If there's
an actual reason not to do this in that case or some other case, then
let's document that reason.  But weird warts in the code that are
justified only by nameless anxiety are not good.

Of course, the prior question is whether we should EVER be doing this.
I realize that something like this MAY be needed for partition-wise
join, but the mission of this patch is not to implement partition-wise
join.  Does anything in this patch series really require this?  If so,
what?  If not, how about we leave it out and refactor it when that
change is really needed for something?

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Tue, Nov 1, 2016 at 1:49 PM, Corey Huinker <corey.huinker@gmail.com> wrote:
> Exactly. This is especially true for date ranges. There's a lot of cognitive
> dissonance in defining the "2014" partition as < '2015-01-01', as was the
> case in Oracle waterfall-style partitioning. That was my reasoning for
> pushing for range-ish syntax as well as form.

Yeah.  That syntax has some big advantages, though.  If we define that
partition as START ('2014-01-01') INCLUSIVE END ('2014-12-31')
INCLUSIVE, there's no way for the system to tell that the there's no
gap between the that ending bound and the starting bound of the 2015
partition, because the system has no domain-specific knowledge that
there is no daylight between 2014-12-31 and 2015-01-01.  So if we
allow things to be specified that way, then people will use that
syntax and then complain when it doesn't perform quite as well as
START ('2014-01-01') END ('2015-01-01').  Maybe the difference isn't
material and maybe we don't care; what do you think?

(I really don't want to get tied up adding a system for adding and
subtracting one to and from arbitrary data types.  Life is too short.
If that requires that users cope with a bit of cognitive dissidence,
well, it's not the first time something like that will have happened.
I have some cognitive dissidence about the fact that creat(2) has no
trailing "e" but truncate(2) does, and moreover the latter can be used
to make a file longer rather than shorter.  But, hey, that's what you
get for choosing a career in computer science.)

>> But who wants that?  People who are doing prefix-based partitioning of
>> their text keys are going to want all of the 'a' things together, and
>> all of the 'b' things in another category.  Same for ranges of
>> floating-point numbers, which are also probably an unlikely candidate
>> for a partitioning key anyway.
>
> /me raises hand.  We have tables with a taxonomy in them where the even data
> splits don't fall on single letter boundaries, and often the single string
> values have more rows than entire letters. In those situations, being able
> to express ['XYZ','XYZ'] is important.  ['XYZ,'XZ') would let 'XYZ1' bleed
> into the partition and ['XYZ','XYZ1') lets in other values, and so I go
> chasing down the non-discrete set rabbit hole.

Hmm.  I have to admit that I hadn't considered the case where you have
a range partitioning scheme but one of the ranges includes only a
single string.  If that's an important use case, that might be a fatal
problem with my proposal.  :-(

> If we're worried about keywords, maybe a BOUNDED '[]' clause?

In the end, keywords are not the defining issue here; the issue is
whether all of this complexity around inclusive and exclusive bounds
carries its weight, and whether we want to be committed to that.

Any other opinions out there?

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Tue, Nov 1, 2016 at 6:49 PM, Corey Huinker <corey.huinker@gmail.com> wrote:
>
> On Tue, Nov 1, 2016 at 12:57 PM, Robert Haas <robertmhaas@gmail.com> wrote:
>> For strings and numeric types that are not integers, there is in
>> theory a loss of power.  If you want a partition that allows very
>> value starting with 'a' plus the string 'b' but not anything after
>> that, you are out of luck.  START ('a') END ('b') INCLUSIVE would have
>> done exactly what you want, but now you need to store the first string
>> that you *don't* want to include in that partition, and what's that?
>> Dunno.  Or similarly if you want to store everything from 1.0 up to
>> and including 2.0 but nothing higher, you can't, really.
> Exactly. This is especially true for date ranges. There's a lot of cognitive
> dissonance in defining the "2014" partition as < '2015-01-01', as was the
> case in Oracle waterfall-style partitioning. That was my reasoning for
> pushing for range-ish syntax as well as form.

OTOH I've seen a lot of people bitten by [2014-01-01,2014-12-31] on
TIMESTAMP intervals.

Everybody remembers december has 31 days, but when we have to do
MONTHLY partitions if you use closed intervals someone always miskeys
the number of days, or forgets wheter a particular year is leap or
not, and when doing it automatically I always have to code it as start
+ 1 month - 1day. In my experience having the non-significant part of
the dates ( days in monthly case, months too in yearly cases ) both 1
and equal in start and end makes it easier to check and identify, and
less error prone.

>> But who wants that?  People who are doing prefix-based partitioning of
>> their text keys are going to want all of the 'a' things together, and
>> all of the 'b' things in another category.  Same for ranges of
>> floating-point numbers, which are also probably an unlikely candidate
>> for a partitioning key anyway.
> /me raises hand.  We have tables with a taxonomy in them where the even data
> splits don't fall on single letter boundaries, and often the single string
> values have more rows than entire letters. In those situations, being able
> to express ['XYZ','XYZ'] is important.  ['XYZ,'XZ') would let 'XYZ1' bleed
> into the partition and ['XYZ','XYZ1') lets in other values, and so I go
> chasing down the non-discrete set rabbit hole.

You just do the classical ( I've had to do it ) closed end || minimum
char ( "XYZ","XYZ\0" in this case ). It is not that difficult as
strings have a global order, the next string to any one is always that
plus the \0, or whatever your minimum is.

The problem is with anything similar to a real number, but then there
I've always opted for half-open interval, as they can cover the line
without overlapping, unlike closed ones.

Anyway, as long as anyone makes sure HALF-OPEN intervals are allowed,
I'm fine ( I do not remember the name, but once had to work with a
system that only allowed closed or open and it was a real PITA.

Francisco Olarte.

On Tue, Nov 1, 2016 at 2:05 PM, Francisco Olarte <folarte@peoplecall.com> wrote:
>> /me raises hand.  We have tables with a taxonomy in them where the even data
>> splits don't fall on single letter boundaries, and often the single string
>> values have more rows than entire letters. In those situations, being able
>> to express ['XYZ','XYZ'] is important.  ['XYZ,'XZ') would let 'XYZ1' bleed
>> into the partition and ['XYZ','XYZ1') lets in other values, and so I go
>> chasing down the non-discrete set rabbit hole.
>
> You just do the classical ( I've had to do it ) closed end || minimum
> char ( "XYZ","XYZ\0" in this case ). It is not that difficult as
> strings have a global order, the next string to any one is always that
> plus the \0, or whatever your minimum is.

In defense of Corey's position, that's not so easy.  First, \0 doesn't
work; our strings can't include null bytes.  Second, the minimum legal
character depends on the collation in use.  It's not so easy to figure
out what the "next" string is, even though there necessarily must be
one.

> The problem is with anything similar to a real number, but then there
> I've always opted for half-open interval, as they can cover the line
> without overlapping, unlike closed ones.
>
> Anyway, as long as anyone makes sure HALF-OPEN intervals are allowed,
> I'm fine ( I do not remember the name, but once had to work with a
> system that only allowed closed or open and it was a real PITA.

I think we're all in agreement that half-open intervals should not
only be allowed, but the default.  The question is whether it's a good
idea to also allow other possibilities.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Tue, Nov 1, 2016 at 7:01 PM, Robert Haas <robertmhaas@gmail.com> wrote:
> In the end, keywords are not the defining issue here; the issue is
> whether all of this complexity around inclusive and exclusive bounds
> carries its weight, and whether we want to be committed to that.
>
> Any other opinions out there?

If it where for me I would opt for just half-open intervals. The only
problem I've ever had with them is when working with FINITE ranges,
i.e., there is no way of expresing the range of 8 bits integer with
half open intervals of 8 bit integers, but I would happily pay that
cost for the benefits of not having people unintentionally make
non-contiguous date/timestamp intervals, which I periodically suffer.

Francisco Olarte.

Robert:

On Tue, Nov 1, 2016 at 7:09 PM, Robert Haas <robertmhaas@gmail.com> wrote:
> In defense of Corey's position, that's not so easy.  First, \0 doesn't
> work; our strings can't include null bytes.  Second, the minimum legal
> character depends on the collation in use.  It's not so easy to figure
> out what the "next" string is, even though there necessarily must be
> one.

I'm aware of that, just wanted to point that it can be done on strings.

> I think we're all in agreement that half-open intervals should not
> only be allowed, but the default.  The question is whether it's a good
> idea to also allow other possibilities.

In my experience, people continuously misuse them. I would specially
like to have them disallowed on timestamp columns ( and other
real-like data, including numeric ). But knowing they cannot do a few
things, and some others are easier with them is enough for allowing
them as an explicit non default for me.

Francisco Olarte.

On Tue, Nov 1, 2016 at 2:11 PM, Corey Huinker <corey.huinker@gmail.com> wrote:
> On Tue, Nov 1, 2016 at 2:01 PM, Robert Haas <robertmhaas@gmail.com> wrote:
>> Yeah.  That syntax has some big advantages, though.  If we define that
>> partition as START ('2014-01-01') INCLUSIVE END ('2014-12-31')
>> INCLUSIVE, there's no way for the system to tell that the there's no
>> gap between the that ending bound and the starting bound of the 2015
>> partition, because the system has no domain-specific knowledge that
>> there is no daylight between 2014-12-31 and 2015-01-01.  So if we
>> allow things to be specified that way, then people will use that
>> syntax and then complain when it doesn't perform quite as well as
>> START ('2014-01-01') END ('2015-01-01').  Maybe the difference isn't
>> material and maybe we don't care; what do you think?
>
> It was a fight I didn't expect to win, and even if we don't get
> [x,x]-expressible partitions, at least we're not in the Oracle
> context-waterfall, where the lower bound of your partition is determined by
> the upper bound of the NEXT partition.

I certainly agree that was a horrible design decision on Oracle's
part.  It's really messy.  If you drop a partition, it changes the
partition constraint for the adjacent partition.  Then you want to add
the partition back, say, but you have to first check whether the
adjacent partition, whose legal range has been expanded, has any
values that are out of bounds.  Which it can't, but you don't know
that, so you have to scan the whole partition.  Aargh!  Maybe this
somehow works out in their system - I'm not an Oracle expert - but I
think it's absolutely vital that we don't replicate it into
PostgreSQL.  (I have some, ahem, first-hand knowledge of how that
works out.)

>> (I really don't want to get tied up adding a system for adding and
>> subtracting one to and from arbitrary data types.  Life is too short.
>> If that requires that users cope with a bit of cognitive dissidence,
>> well, it's not the first time something like that will have happened.
>> I have some cognitive dissidence about the fact that creat(2) has no
>> trailing "e" but truncate(2) does, and moreover the latter can be used
>> to make a file longer rather than shorter.  But, hey, that's what you
>> get for choosing a career in computer science.)
>
> That noise your heard was the sound of my dream dying.

Well, I'm not sure we've exactly reached consensus here, and you're
making me feel like I just kicked a puppy.  However, my goal here is
not to kill your dream but to converge on a committable patch as
quickly as possible.  Adding increment/decrement operators to every
discrete(-ish) type we have is not the way to accomplish that.  To the
contrary, that's just about guaranteed to make this patch take an
extra release cycle to finish.  Now, that does not necessarily mean we
can't keep the INCLUSIVE/EXCLUSIVE stuff -- after all, Amit has
already written it -- but then we have to live with the fact that
+1/-1 based optimizations and matching are not going to be there.
Whether it's still worth having fully-open and fully-closed ranges on
general principle -- and whether the lack of such optimizations
changes the calculus -- is what we are now debating.

More votes welcome.  Right now I count 2 votes for keeping the
inclusive-exclusive stuff (Amit, Corey) and two for nuking it from
orbit (Robert, Francisco).  I'm not going to put my foot down on this
point against a contrary consensus, so please chime in.  Thanks.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

<div dir="ltr"><div class="gmail_extra"><br /><div class="gmail_quote">On Tue, Nov 1, 2016 at 2:24 PM, Robert Haas
<spandir="ltr"><<a href="mailto:robertmhaas@gmail.com" target="_blank">robertmhaas@gmail.com</a>></span>
wrote:<br/><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div
class="a3saXjCH m1582122e59b1b8eb" id=":2dm">Well, I'm not sure we've exactly reached consensus here, and you're<br />
makingme feel like I just kicked a puppy. </div></blockquote></div><br />It was hyperbole. I hope you found it as funny
toread as I did to write. This is a great feature and I'm not going to make "perfect" the enemy of "good".</div></div> 

On 2016/11/02 2:53, Robert Haas wrote:
> On Fri, Oct 28, 2016 at 3:53 AM, Amit Langote
> <Langote_Amit_f8@lab.ntt.co.jp> wrote:
>>  [ new patches ]
> 
> Reviewing 0005:
> 
> Your proposed commit messages says this:
> 
>> If relation is the target table (UPDATE and DELETE), flattening is
>> done regardless (scared to modify inheritance_planner() yet).

I should have explained my thinking behind why I ended up not handling the
result relation case:

While set_append_rel_size() can be safely invoked recursively on the roots
of partition sub-trees, I was not quite sure if inheritance_planner() is
amenable to such recursive invocation.  While the former is relatively
straightforward baserel processing, the latter is not-so-straightforward
transformation of the whole query for every target child relation of the
target parent.

> In the immortal words of Frank Herbert: “I must not fear. Fear is the
> mind-killer. Fear is the little-death that brings total obliteration.
> I will face my fear. I will permit it to pass over me and through me.
> And when it has gone past I will turn the inner eye to see its path.
> Where the fear has gone there will be nothing. Only I will remain.”
> 
> In other words, I'm not going to accept fear as a justification for
> randomly-excluding the target-table case from this code.  If there's
> an actual reason not to do this in that case or some other case, then
> let's document that reason.  But weird warts in the code that are
> justified only by nameless anxiety are not good.

Perhaps the above comment expands a little on the actual reason.  Though I
guess it still amounts to giving up on doing a full analysis of whether
recursive processing within inheritance_planner() is feasible.

I think we could just skip this patch as long as a full investigation into
inheritance_planner() issues is not done.  It's possible that there might
be other places in the planner code which are not amenable to the proposed
multi-level AppendRelInfos.  Ashutosh had reported one [1], wherein
lateral joins wouldn't work with multi-level partitioned table owing to
the multi-level AppendRelInfos (the patch contains a hunk to address that).

> Of course, the prior question is whether we should EVER be doing this.
> I realize that something like this MAY be needed for partition-wise
> join, but the mission of this patch is not to implement partition-wise
> join.  Does anything in this patch series really require this?  If so,
> what?  If not, how about we leave it out and refactor it when that
> change is really needed for something?

Nothing *requires* it as such.  One benefit I see is that exclusion of the
root of some partition sub-tree means the whole sub-tree is excluded in
one go.  Currently, because of the flattening, every relation in that
sub-tree would be excluded separately, needlessly repeating the expensive
constraint exclusion proof. But I guess that's just an optimization.

Thanks,
Amit

[1]
https://www.postgresql.org/message-id/CAFjFpReZF34MDbY95xoATi0xVj2mAry4-LHBWVBayOc8gj%3Diqg%40mail.gmail.com

On 2016/11/02 2:34, Robert Haas wrote:
> On Fri, Oct 28, 2016 at 3:53 AM, Amit Langote
> <Langote_Amit_f8@lab.ntt.co.jp> wrote:
>> [ new patches ]
> 
> Reviewing 0006:

Thanks for the review!

> This patch seems scary.  I sort of assumed from the title -- "Teach a
> few places to use partition check quals." -- that this was an optional
> thing, some kind of optimization from which we could reap further
> advantage once the basic infrastructure was in place.  But it's not
> that at all.  It's absolutely necessary that we do this, or data
> integrity is fundamentally compromised.  How do we know that we've
> found all of the places that need to be taught about these new,
> uncatalogued constraints?

Making this a separate commit from 0003 was essentially to avoid this
getting lost among all of its other changes.  In fact, it was to bring to
notice for closer scrutiny whether all the sites in the backend code that
are critical for data integrity in face of the implicit partition
constraints are being informed about those constraints.

> I'm feeling fairly strongly like you should rewind and make the
> partitioning constraints normal catalogued constraints.  That's got a
> number of advantages, most notably that we can be sure they will be
> properly enforced by the entire system (modulo existing bugs, of
> course).  Also, they'll show up automatically in tools like psql's \d
> output, pgAdmin, and anything else that is accustomed to being able to
> find constraints in the catalog.  We do need to make sure that those
> constraints can't be dropped (or altered?) inappropriately, but that's
> a relatively small problem.  If we stick with the design you've got
> here, every client tool in the world needs to be updated, and I'm not
> seeing nearly enough advantage in this system to justify that kind of
> upheaval.

As for which parts of the system need to know about these implicit
partition constraints to *enforce* them for data integrity, we could say
that it's really just one site - ExecConstraints() called from
ExecInsert()/ExecUpdate().

Admittedly, the current error message style as in this patch exposes the
implicit constraint approach to a certain criticism: "ERROR:  new row
violates the partition boundary specification of \"%s\"".  It would say
the following if it were a named constraint: "ERROR: new row for relation
\"%s\" violates check constraint \"%s\""

For constraint exclusion optimization, we teach get_relation_constraints()
to look at these constraints.  Although greatly useful, it's not the case
of being absolutely critical.

Beside the above two cases, there is bunch of code (relcache, DDL) that
looks at regular constraints, but IMHO, we need not let any of that code
concern itself with the implicit partition constraints.  Especially, I
wonder why the client tools should want the implicit partitioning
constraint to be shown as a CHECK constraint?  As the proposed patch 0004
(psql) currently does, isn't it better to instead show the partition
bounds as follows?

+CREATE TABLE part_b PARTITION OF parted (
+    b WITH OPTIONS NOT NULL DEFAULT 1 CHECK (b >= 0),
+    CONSTRAINT check_a CHECK (length(a) > 0)
+) FOR VALUES IN ('b');

+\d part_b
+         Table "public.part_b"
+ Column |  Type   |     Modifiers
+--------+---------+--------------------
+ a      | text    |
+ b      | integer | not null default 1
+Partition of: parted FOR VALUES IN ('b')
+Check constraints:
+    "check_a" CHECK (length(a) > 0)
+    "part_b_b_check" CHECK (b >= 0)

Needless to say, that could save a lot of trouble thinking about
generating collision-free names of these constraints, their dependency
handling, unintended altering of these constraints, pg_dump, etc.

> In fact, as far as I can see, the only advantage of this approach is
> that when the insert arrives through the parent and is routed to the
> child by whatever tuple-routing code we end up with (I guess that's
> what 0008 does), we get to skip checking the constraint, saving CPU
> cycles.  That's probably an important optimization, but I don't think
> that putting the partitioning constraint in the catalog in any way
> rules out the possibility of performing that optimization.  It's just
> that instead of having the partitioning excluded-by-default and then
> sometimes choosing to include it, you'll have it included-by-default
> and then sometimes choose to exclude it.

Hmm, doesn't it seem like we would be making *more* modifications to the
existing code (backend or otherwise) to teach it about excluding the
implicit partition constraints than the other way around?  The other way
around being to modify the existing code to include the implicit
constraints which is what this patch is about.

Having said all that, I am open to switching to the catalogued partition
constraints if the arguments I make above in favor of this patch are not
all that sound.

Thanks,
Amit

On 2016/11/02 16:41, Amit Langote wrote:
> Having said all that, I am open to switching to the catalogued partition
> constraints if the arguments I make above in favor of this patch are not
> all that sound.

One problem I didn't immediately see a solution for if we go with the
catalogued partition constraints is how do we retrieve a relation's
partition constraint?  There are a couple of cases where that becomes
necessary.  Consider that we are adding a partition to a partitioned table
that is itself partition.  In this case, the new partition's constraint
consists of whatever we generate from its FOR VALUES specification *ANDed*
with the parent's constraint.  We must somehow get hold of the latter.
Which of the parent's named check constraints in the pg_constraint catalog
is supposed to be the partition constraint?  With the uncatalogued
partition constraints approach, we simply generate it from the parent's
pg_class.relpartbound (or we might get the relcache copy of the same
stored in rd_partcheck).  Hmm.

Thanks,
Amit

On 2016/11/02 2:44, Robert Haas wrote:
> On Fri, Oct 28, 2016 at 3:53 AM, Amit Langote
> <Langote_Amit_f8@lab.ntt.co.jp> wrote:
>>> Insisting that you can't drop a child without detaching it first seems
>>> wrong to me.  If I already made this comment and you responded to it,
>>> please point me back to whatever you said.  However, my feeling is
>>> this is flat wrong and absolutely must be changed.
>>
>> I said the following [1]:
>>
>> | Hmm, I don't think I like this.  Why should it be necessary to detach
>> | a partition before dropping it?  That seems like an unnecessary step.
>>
>> I thought we had better lock the parent table when removing one of its
>> partitions and it seemed a bit odd to lock the parent table when dropping
>> a partition using DROP TABLE?  OTOH, with ALTER TABLE parent DETACH
>> PARTITION, the parent table is locked anyway.
> 
> That "OTOH" part seems like a pretty relevant point.
> 
> Basically, I think people expect to be able to say "DROP THINGTYPE
> thingname" or at most "DROP THINGTYPE thingname CASCADE" and have that
> thing go away.  I'm opposed to anything which requires some other
> series of steps without a very good reason, and possible surprise
> about the precise locks that the command requires isn't a good enough
> reason from my point of view.

OK, I changed things so that DROP TABLE a_partition no longer complains
about requiring to detach first.  Much like how index_drop() locks the
parent table ('parent' in a different sense, of course) and later
invalidates its relcache, heap_drop_with_catalog(), if the passed in relid
is a partition, locks the parent table using AccessExclusiveLock, performs
its usual business, and finally invalidates the parent's relcache before
closing it without relinquishing the lock.  Does that sounds sane?  One
downside is that if the specified command is DROP TABLE parent CASCADE,
the above described invalidation is a waste of cycles because the parent
will be dropped anyway after all the partitions are dropped.

Thanks,
Amit

El 2016-10-28 07:53, Amit Langote escribió:

> @@ -6267,6 +6416,12 @@ ATAddForeignKeyConstraint(AlteredTableInfo *tab, 
> Relation rel,
>       * Validity checks (permission checks wait till we have the column
>       * numbers)
>       */
> +    if (pkrel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
> +        ereport(ERROR,
> +                (errcode(ERRCODE_WRONG_OBJECT_TYPE),
> +                 errmsg("cannot reference relation \"%s\"", 
> RelationGetRelationName(pkrel)),
> +                 errdetail("Referencing partitioned tables in foreign key 
> constraints is not supported.")));

Is there a plan for fixing this particular limitation?  It's a pretty 
serious problem for users,
and the suggested workaround (to create a separate non-partitioned table 
which carries only the PK
columns which is updated by triggers, and direct the FKs to it instead 
of to the partitioned table)
is not only a very ugly one, but also very slow.

On Wed, Nov 2, 2016 at 6:47 AM, Amit Langote
<Langote_Amit_f8@lab.ntt.co.jp> wrote:
> OK, I changed things so that DROP TABLE a_partition no longer complains
> about requiring to detach first.  Much like how index_drop() locks the
> parent table ('parent' in a different sense, of course) and later
> invalidates its relcache, heap_drop_with_catalog(), if the passed in relid
> is a partition, locks the parent table using AccessExclusiveLock, performs
> its usual business, and finally invalidates the parent's relcache before
> closing it without relinquishing the lock.  Does that sounds sane?

Yes.

> One
> downside is that if the specified command is DROP TABLE parent CASCADE,
> the above described invalidation is a waste of cycles because the parent
> will be dropped anyway after all the partitions are dropped.

I don't think that's even slightly worth worrying about.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Thu, Nov 3, 2016 at 7:46 AM,  <alvherre@alvh.no-ip.org> wrote:
> El 2016-10-28 07:53, Amit Langote escribió:
>> @@ -6267,6 +6416,12 @@ ATAddForeignKeyConstraint(AlteredTableInfo *tab,
>> Relation rel,
>>          * Validity checks (permission checks wait till we have the column
>>          * numbers)
>>          */
>> +       if (pkrel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
>> +               ereport(ERROR,
>> +                               (errcode(ERRCODE_WRONG_OBJECT_TYPE),
>> +                                errmsg("cannot reference relation
>> \"%s\"", RelationGetRelationName(pkrel)),
>> +                                errdetail("Referencing partitioned tables
>> in foreign key constraints is not supported.")));
>
> Is there a plan for fixing this particular limitation?  It's a pretty
> serious problem for users,
> and the suggested workaround (to create a separate non-partitioned table
> which carries only the PK
> columns which is updated by triggers, and direct the FKs to it instead of to
> the partitioned table)
> is not only a very ugly one, but also very slow.

If you have two compatibly partitioned tables, and the foreign key
matches the partitioning keys, you could implement a foreign key
between the two tables as a foreign key between each pair of matching
partitions.  Otherwise, isn't the only way to handle this a global
index?

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Tue, Nov 1, 2016 at 2:36 PM, Corey Huinker <corey.huinker@gmail.com> wrote:
> On Tue, Nov 1, 2016 at 2:24 PM, Robert Haas <robertmhaas@gmail.com> wrote:
>> Well, I'm not sure we've exactly reached consensus here, and you're
>> making me feel like I just kicked a puppy.
>
> It was hyperbole. I hope you found it as funny to read as I did to write.
> This is a great feature and I'm not going to make "perfect" the enemy of
> "good".

Oh, OK.  Sorry, the tone did not transfer to my brain in the way you
intended it - I thought you were actually upset.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Wed, Nov 2, 2016 at 3:41 AM, Amit Langote
<Langote_Amit_f8@lab.ntt.co.jp> wrote:
> As for which parts of the system need to know about these implicit
> partition constraints to *enforce* them for data integrity, we could say
> that it's really just one site - ExecConstraints() called from
> ExecInsert()/ExecUpdate().

Well, that's a slightly optimistic view of the situation, because the
issue is whether ExecConstraints() is going to get called in the first
place.  But now that I look at it, there are only a handful of
callers, so maybe it's not so bad.

> Admittedly, the current error message style as in this patch exposes the
> implicit constraint approach to a certain criticism: "ERROR:  new row
> violates the partition boundary specification of \"%s\"".  It would say
> the following if it were a named constraint: "ERROR: new row for relation
> \"%s\" violates check constraint \"%s\""
>
> For constraint exclusion optimization, we teach get_relation_constraints()
> to look at these constraints.  Although greatly useful, it's not the case
> of being absolutely critical.
>
> Beside the above two cases, there is bunch of code (relcache, DDL) that
> looks at regular constraints, but IMHO, we need not let any of that code
> concern itself with the implicit partition constraints.  Especially, I
> wonder why the client tools should want the implicit partitioning
> constraint to be shown as a CHECK constraint?  As the proposed patch 0004
> (psql) currently does, isn't it better to instead show the partition
> bounds as follows?
>
> +CREATE TABLE part_b PARTITION OF parted (
> +       b WITH OPTIONS NOT NULL DEFAULT 1 CHECK (b >= 0),
> +       CONSTRAINT check_a CHECK (length(a) > 0)
> +) FOR VALUES IN ('b');

Good point.

> +\d part_b
> +         Table "public.part_b"
> + Column |  Type   |     Modifiers
> +--------+---------+--------------------
> + a      | text    |
> + b      | integer | not null default 1
> +Partition of: parted FOR VALUES IN ('b')
> +Check constraints:
> +    "check_a" CHECK (length(a) > 0)
> +    "part_b_b_check" CHECK (b >= 0)
>
> Needless to say, that could save a lot of trouble thinking about
> generating collision-free names of these constraints, their dependency
> handling, unintended altering of these constraints, pg_dump, etc.

Well, that's certainly true, but those problems don't strike me as so
serious that they couldn't be solved with a reasonable amount of
labor.

>> In fact, as far as I can see, the only advantage of this approach is
>> that when the insert arrives through the parent and is routed to the
>> child by whatever tuple-routing code we end up with (I guess that's
>> what 0008 does), we get to skip checking the constraint, saving CPU
>> cycles.  That's probably an important optimization, but I don't think
>> that putting the partitioning constraint in the catalog in any way
>> rules out the possibility of performing that optimization.  It's just
>> that instead of having the partitioning excluded-by-default and then
>> sometimes choosing to include it, you'll have it included-by-default
>> and then sometimes choose to exclude it.
>
> Hmm, doesn't it seem like we would be making *more* modifications to the
> existing code (backend or otherwise) to teach it about excluding the
> implicit partition constraints than the other way around?  The other way
> around being to modify the existing code to include the implicit
> constraints which is what this patch is about.

I'm not sure which way is actually going to be more code modification,
but I guess you've persuaded me that the way you have it is
reasonable, so let's stick with that.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Apologies if I've made some of these comments before and/or missed
comments you've made on these topics.  The size of this patch set is
so large that it's hard to keep track of everything.

Re-reviewing 0001:

+       indicate which table columns are used as partition key.  For example,

s/are used as/are part of the/

+       third table columns make up the partition key.  A zero in this array
+       indicates that the corresponding partition key column is an expression
+       over the table columns, rather than a simple column reference.

I think you can leave out "over the table columns".

+      columns or expressions forms the <firstterm>partitioning key</firstterm>

s/forms/form/

+      The table itself is empty.  A data row inserted into the table is routed

s/The table/The partitioned table/

+     * Anything mentioned in the expressions.  We must ignore the column
+     * references which will count as self-dependency items; in this case,
+     * the depender is the table itself (there is no such thing as partition
+     * key object).

"depender" is not really a word, and the parenthetical note isn't very
clear.  Maybe: We must ignore the column references, which will depend
on the table itself; there is no separate partition key object.

+    heap_close(pg_partitioned_table, RowExclusiveLock);

It seems like it would be safe to do this right after
CatalogUpdateIndexes(pg_partitioned_table, tuple), and I'd recommend
that you do.  Not for performance or anything, but just to keep
related code together.
/* * Resolve possibly-defaulted operator class specification */
-static Oid
+OidGetIndexOpClass(List *opclass, Oid attrType,

Perhaps we should rename this function to ResolveOpClass, since it's
now going to be used for both indexes and partitioning keys.

+ * Sets *is_expr if attnum is found to be referenced in some partition key
+ * expression.

is_expr doesn't seem quite as clear as, say, used_by_expr or used_in_expr.

Also, the specification for this function doesn't seem to be very
clear about what this is supposed to do if the same column is both an
explicit partitioning column and also used in an expression, and the
code looks like it'll return with *is_expr set based on whichever use
it encounters first.  If that's intended behavior, maybe add a comment
like: It's possible for a column to be used both directly and as part
of a partition key expression; if that happens, *is_expr may end up as
either true or false.  That's OK for current uses of this function,
because *is_expr is only used to tailor the error message text.

+            if (is_expr)
+                *is_expr = false;
+            if (attnum == partattno)
+                return true;

I think you should adjust this (and the other bit in the same
function) so that you don't set *is_expr until you're committed to
returning.

+            index = -1;
+            while ((index = bms_next_member(expr_attrs, index)) > 0)
+            {
+                AttrNumber attno = index + FirstLowInvalidHeapAttributeNumber;
+
+                if (attno == attnum)
+                    return true;
+            }

How about bms_is_member(expr_attrs, attnum -
FirstLowInvalidHeapAttributeNumber), instead of looping?

+                 errmsg("cannot reference relation \"%s\"",
RelationGetRelationName(pkrel)),
+                 errdetail("Referencing partitioned tables in foreign
key constraints is not supported.")));

I think you could get rid of the errdetail and just have the error
message be "cannot reference partitioned table \"%s\"".

+                         errmsg("column \"%s\" appears twice in
partition key", pelem->name),

It could be there three times!  How about column \"%s\" appears more
than once in partition key?  (I see that you seem to have adapted this
from some code in parse_utilcmd.c, which perhaps should also be
adjusted, but that's a job for another patch.)

+            /*
+             * Strip any top-level COLLATE clause.  This ensures that we treat
+             * "x COLLATE y" and "(x COLLATE y)" alike.
+             */

But you don't, right?  Unless I am confused, which is possible, the
latter COLLATE will be ignored, while the former one will set the
collation to be used in the context of partitioning comparisons.

Re-reviewing 0002:

+       if (fout->remoteVersion >= 100000)
+       {
+               PQExpBuffer acl_subquery = createPQExpBuffer();
+               PQExpBuffer racl_subquery = createPQExpBuffer();
+               PQExpBuffer initacl_subquery = createPQExpBuffer();
+               PQExpBuffer initracl_subquery = createPQExpBuffer();
+
+               PQExpBuffer attacl_subquery = createPQExpBuffer();
+               PQExpBuffer attracl_subquery = createPQExpBuffer();
+               PQExpBuffer attinitacl_subquery = createPQExpBuffer();
+               PQExpBuffer attinitracl_subquery = createPQExpBuffer();

It seems unnecessary to repeat all of this.  The only differences
between the 10000 version and the 9600 version are:

60,61c60
<                           "AS changed_acl, "
<                           "CASE WHEN c.relkind = 'P' THEN
pg_catalog.pg_get_partkeydef(c.oid) ELSE NULL END AS partkeydef "
---
>                           "AS changed_acl "
73c72
<                    "WHERE c.relkind in ('%c', '%c', '%c', '%c',
'%c', '%c', '%c') "
---
>                    "WHERE c.relkind in ('%c', '%c', '%c', '%c', '%c', '%c') "
87,88c86
<                           RELKIND_MATVIEW, RELKIND_FOREIGN_TABLE,
<                           RELKIND_PARTITIONED_TABLE);
---
>                           RELKIND_MATVIEW, RELKIND_FOREIGN_TABLE);

But none of that is really a problem.  Sure, the 'P' case will never
arise in 9.6, but so what?  I'd really like to not keep duplicating
these increasingly-complex hunks of code if we can find some way to
avoid that.
       /* find all the inheritance information */

-       appendPQExpBufferStr(query, "SELECT inhrelid, inhparent FROM
pg_inherits");
+       appendPQExpBufferStr(query,
+                                                "SELECT inhrelid, inhparent "
+                                                "FROM pg_inherits "
+                                                "WHERE inhparent NOT
IN (SELECT oid FROM pg_class WHERE relkind = 'P')");

I think you need to update the comment.  "Find inheritance
information, excluding implicit inheritance via partitioning.  We're
not interested in that case because $REASON."

Re-reviewing 0003:

+     <para>
+      If this table is a partition, one cannot perform <literal>DROP
NOT NULL</>
+      on a column if it is marked not null in the parent table.
+     </para>

This would, presumably, also be true for inheritance.  I think we
could just leave this out.

+      as partition of the target table.  The partition bound specification must

s/as partition/as a partition/

+      correspond to the partitioning method and partitioning key of the target

I think that in most places were are referring to the "partitioning
method" (with ing) but the "partition key" (without ing). Let's try to
be consistent.

+      table.  The table to be attached must have all the columns as the target
+      table and no more; moreover, the column types must also match.  Also, it
+      must have all the matching constraints as the target table.

s/all the columns/all of the same columns/

The second sentence doesn't seem quite grammatical.  And why would
that be true anyway?  Partitions can have extra constraints, and if
they lack constraints that are present on the partitioned table, those
constraints will be added and validated, right?

+      A full table scan is performed on the table being attached to check that
+      existing rows in the table are within the specified partition bound.
+      If it is known in advance that no partition bound violating rows are
+      present in the table, the above scan can be skipped by specifying the
+      <literal>NO VALIDATE</> option.
+      Default behavior is to perform the scan, as if the
<literal>VALIDATE</literal>
+      option were specified.

I don't think it's OK to allow the partition to be added if it
contains rows that might not be valid.  We are generally vary wary
about adding options that can cause data integrity failures and I
think we shouldn't start here, either.  On the other hand, it's also
not desirable for adding a partition to take O(n) time in all cases.
So what would be nice is if the new partition could self-certify that
contains no problematic rows by having a constraint that matches the
new partitioning constraint.  Then we can skip the scan if that
constraint is already present.

+      inherited columns.  One can also specify table constraints, in addition

Delete comma.

+      to those inherited from the parent.  If a check constraint with the name
+      matching one of the parent's constraint is specified, it is merged with
+      the latter, provided the specified condition is same.

Doesn't that effectively delete the merged constraint?

Suggest: "If the parent already has a check constraint with the same
name as a constraint specified for the child, the conditions must be
the same."

+) FOR VALUES IN ('los angeles', 'san fransisco');

That's not you you spell San Francisco.

+   Create partition of a list partitioned table that itself is further
+   partitioned and then create its partition:

s/itself is/is itself/
s/then create its partition/then add a partition to it/

+                               if (!is_local || con->coninhcount == 0)
+                                       con->coninhcount++;

I would think that you could get rid of the "if" and just say
con->coninhcount = 1.  It seems to me (and perhaps the comment could
say this) that for a partitioned table, we can simplify the handling
of coninhcount and conislocal.  Since partitioning hierarchies don't
allow multiple inheritance, any inherited constraint must be inherited
exactly once.  Since a partition cannot have inheritance children --
except by being partitioned itself -- there is no point in tracking
conislocal, so we always store it as false.

+
+void
+StorePartitionBound(Relation rel, Node *bound)

Header comment, please!

+       (void) SysCacheGetAttr(RELOID, tuple, Anum_pg_class_relpartbound,
+                                                  &isnull);
+       Assert(isnull);

We try not to do unnecessary work in non-assert-enabled builds solely
for the benefit of assert-enabled builds.  We also try not to end up
with variables that are only used in assert-enabled builds but not
marked PG_USED_FOR_ASSERTS_ONLY, because those tend to cause compiler
warnings.  I'm not sure an compiler would be smart enough to warn
about this, but I suggest adding an #ifdef USE_ASSERT_CHECKING with a
block inside where the offending variable is declared.  Actually, I
think you need to move a bit more code.  Hmm.  Something like:

#ifdef USE_ASSERT_CHECKING   {       Form_pg_class classForm;       bool isnull;
      classForm = (Form_pg_class) GETSTRUCT(tuple);      Assert(!classForm->relispartition);
      (void) SysCacheGetAttr(RELOID, tuple, Anum_pg_class_relpartbound,
&isnull);     Assert(isnull);   }
 
#endif

+ * are same in common cases, of which we only store one.

"and we only store one of them."

+                                                       /*
+                                                        * Never put a
null into the values array, flag
+                                                        * instead for
the code further down below where
+                                                        * we
construct the actual relcache struct.
+                                                        */
+
found_null_partition = true;
+
null_partition_index = i;

How about we elog(ERROR, ...) if found_null_partition is already set?

+                               foreach(cell, non_null_values)
+                               {
+                                       PartitionListValue      *src =
lfirst(cell);
+
+                                       all_values[i] = (PartitionListValue *)
+        palloc(sizeof(PartitionListValue));
+                                       all_values[i]->value =
datumCopy(src->value,
+                                 key->parttypbyval[0],
+                                 key->parttyplen[0]);
+                                       all_values[i]->index = src->index;
+                                       i++;
+                               }

Why do we need to datumCopy() here when we just did that in the
previous loop?  Why did the previous loop need to datumCopy(), anyway?Can't we just pass the same datums around?  I
understandthat you
 
need to copy the data into the correct context at the end, but doing
two copies prior to that seems excessive.

+get_leaf_partition_oids(Oid relid, int lockmode)

Instead of implementing this, can you just use find_all_inheritors()?

+               /*
+                * Is lower_val = upper_val?
+                */
+               if (lower_val && upper_val)

So, that comment does not actually match that code.  That if-test is
just checking whether both bounds are finite.  What I think you should
be explaining here is that if lower_val and upper_val are both
non-infinite, and if the happen to be equal, we want to emit an
equality test rather than a >= test plus a <= test because $REASON.

+       operoid = get_opfamily_member(key->partopfamily[col],
+
key->parttypid[col],
+
key->parttypid[col],
+                                                                 strategy);
+       if (!OidIsValid(operoid))
+       {
+               operoid = get_opfamily_member(key->partopfamily[col],
+  key->partopcintype[col],
+  key->partopcintype[col],
+  strategy);
+               *need_relabel = true;
+       }

There's a comment explaining THAT you do this ("Use either the column
type as the operator datatype or opclass's declared input type.") but,
to repeat a complaint that I've made often, nothing explaining why.
In this particular case, what's not clear - in my opinion - is why you
need to try two different possibilities, and why at least one of those
possibilities is guaranteed to work.  I gather that if the opfamily
doesn't contain an operator for the actual type of the partitioning
column, you think it will certainly contain one for the input type of
the operator class (which seems right), and that the input type of the
operator class will be binary-compatible with the type of the
partitioning column (which is less-obviously true, and needs some
explanation).

I also think that this function should elog(ERROR, ...) if by any
chance the second get_opfamily_member() call also fails. Otherwise the
error might occur quite a bit downstream and be hard to understand.

+       ReleaseSysCache(tuple);
+       heap_close(parent, AccessShareLock);

I think you had better not release the lock here - i.e. pass NoLock.
We don't release locks on tables until transaction commit, except for
catalog tables.  This also comes up in, at least,
transformPartitionOf().

+    PartitionRangeBound *b1 = (*(PartitionRangeBound *const *) a);
+    PartitionRangeBound *b2 = (*(PartitionRangeBound *const *) b);
+       PartitionKey key = (PartitionKey) arg;
+
+       return partition_rbound_cmp(key, b1, b2);

Whitespace.

+        * as partition and schema consists of columns definitions corresponding

the schema consists

-               if (recurse)
+               /* Force inheritance recursion, if partitioned table. */
+               if (recurse || rel->rd_rel->relkind ==
RELKIND_PARTITIONED_TABLE)

I would instead error out if ONLY is specified for a partitioned table.

+               /*
+                * If the table is source table of ATTACH PARTITION
command, following
+                * check is unnecessary.
+                */

As usual, comment should say WHY.

+                       if (partqualstate && !ExecQual(partqualstate,
econtext, true))
+                               ereport(ERROR,
+
(errcode(ERRCODE_CHECK_VIOLATION),
+                                                errmsg("child table
contains a row violating partition bound specification")));

Why not mimic the existing phrasing?  "partition constraint is
violated by some row"

What happens if you try to attach a table as a partition of itself or
one of its ancestors?

-                               errmsg("column \"%s\" in child table
must be marked NOT NULL",
-                                          attributeName)));
+                                                errmsg("column \"%s\"
in child table must be marked NOT NULL",
+
attributeName)));

Whitespace-only hunk; revert.  You cannot fight the power of pgindent.

+                                errmsg("cannot attach table that is a
inheritance child as partition")));

an inheritance child

+               errmsg("cannot attach a temporary relation of another
session as partition ")));

Extra space.

+                                        errdetail("Table being
attached should contain only the columns"
+                                                          " present
in parent.")));

Suggest: "New partition should contain only..."

Also, don't break error messages into multiple strings.  Make it one
long string and let pgindent deal.

+                       | FOR VALUES START '(' range_datum_list ')' lb_inc
+                                                END_P '('
range_datum_list ')' ub_inc

Just a random idea.  Would for VALUES FROM ( ... ) TO ( ... ) be more
idiomatic than START and END?

+static void
+transformPartitionOf(CreateStmtContext *cxt, Node *bound)
+{
+       TupleDesc       tupdesc;
+       int                     i;
+       RangeVar   *part = cxt->relation;
+       RangeVar   *partof = linitial(cxt->inhRelations);
+       Relation        parentRel;
+
+       parentRel = heap_openrv(partof, AccessShareLock);
+

If there's anyway that we might do heap_openrv() on the same RangeVar
at multiple places in the code, it presents security and integrity
hazards because the referent of that RangeVar might change in the
meantime.  I suspect there is such a risk here - won't we need to open
the relation again when we actually want to do the operation?

Also, we should never acquire a lower-level lock on a relation and
then, further downstream, acquire a higher-level lock on the same
object.  To do so creates a deadlock hazard.  That seems like it might
be a problem here, too.

+                               /*if (ldatum->infinite && rdatum->infinite)
+                                       ereport(ERROR,
+
(errcode(ERRCODE_INVALID_TABLE_DEFINITION),
+                                                errmsg("both START
and END datum for column \"%s\" cannot be UNBOUNDED",
+                                                               colname),
+
parser_errposition(cxt->pstate, rdatum->location)));*/

Commented-out code is bad.

+                       if (list_length(spec->lowerdatums) > partnatts)
+                               ereport(ERROR,
+
(errcode(ERRCODE_INVALID_TABLE_DEFINITION),
+                                        errmsg("START has more values
than the length of the partition key"),
+
parser_errposition(cxt->pstate,
+
exprLocation(list_nth(spec->lowerdatums,
+ list_length(spec->lowerdatums) - 1)))));
+                       else if (list_length(spec->lowerdatums) < partnatts)
+                               ereport(ERROR,
+
(errcode(ERRCODE_INVALID_TABLE_DEFINITION),
+                                        errmsg("START has fewer
values than the length of the partition key"),
+
parser_errposition(cxt->pstate,
+
exprLocation(list_nth(spec->lowerdatums,
+ list_length(spec->lowerdatums) - 1)))));

It would be worth looking for a way to unify these cases.  Like "START
must specify exactly one value per partitioning column".

+                * Same oids? No mind order - in the list case, it
matches the order
+                * in which partition oids are returned by a
pg_inherits scan, whereas
+                * in the range case, they are in order of ranges of individual
+                * partitions.  XXX - is the former unsafe?

Probably.  It likely depends on the physical ordering of tuples in the
table, which can change.

+ * BoundCollection encapsulates a set of partition bounds of either physical
+ * or logical relations.  It is associated with a partitioned relation of
+ * which the aforementioned relations are partitions.

"physical or logical relations" is unfamiliar terminology.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 28 October 2016 at 02:53, Amit Langote <Langote_Amit_f8@lab.ntt.co.jp> wrote:
>
> Please find attached the latest version of the patches

Hi,

I started to review the functionality of this patch, so i applied all
9 patches. After that i found this warning, which i guess is because
it needs a cast.

After that, i tried a case that i think is important: to partition an
already existing table. Because there is no ALTER TABL SET PARTITION
or something similar (which i think makes sense because such a command
would need to create the partitions and move the rows to follow the
rule that there is no rows in a parent table).

So, what i tried was:

1) rename original table
2) create a new partitioned table with old name
3) attach old table as a partition with bounds outside normal bounds
and no validate

the idea is to start attaching valid partitions and delete and insert
rows from the invalid one (is there a better way of doing that?), that
will allow to partition a table easily.
So far so good, until i decided points 1 to 3 should happen inside a
transaction to make things transparent to the user.

Attached is script that shows the failure when trying it:

script 1 (failing_test_1.sql) fails the assert
"Assert(RelationGetPartitionKey(parentRel) != NULL);" in
transformAttachPartition() at src/backend/parser/parse_utilcmd.c:3164

After that i tried the same but with an already partitioned (via
inheritance) table and got this (i did this first without a
transaction, doing it with a transaction will show the same failure as
before):

script 2 (failing_test_2.sql) fails the assert
"Assert(childrte->relkind == RELKIND_PARTITIONED_TABLE);" in
expand_inherited_rte_internal() at
src/backend/optimizer/prep/prepunion.c:1551

PS: i don't like the START END syntax but i don't have any ideas
there... except, there was a reason not to use expressions (like a
CHECK constraint?)

--
Jaime Casanova                      www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On 7 November 2016 at 12:15, Jaime Casanova
<jaime.casanova@2ndquadrant.com> wrote:
> On 28 October 2016 at 02:53, Amit Langote <Langote_Amit_f8@lab.ntt.co.jp> wrote:
>>
>> Please find attached the latest version of the patches
>
> Hi,
>
> I started to review the functionality of this patch, so i applied all
> 9 patches. After that i found this warning, which i guess is because
> it needs a cast.
>

oh! i forgot the warning
"""
partition.c: In function ‘get_qual_for_list’:
partition.c:1159:6: warning: assignment from incompatible pointer type
   or = makeBoolExpr(OR_EXPR, list_make2(nulltest2, opexpr), -1);
      ^
"""

attached a list of the warnings that my compiler give me (basically
most are just variables that could be used uninitialized)

--
Jaime Casanova                      www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Hi Jaime,

On 2016/11/08 2:24, Jaime Casanova wrote:
> On 7 November 2016 at 12:15, Jaime Casanova
> <jaime.casanova@2ndquadrant.com> wrote:
>> On 28 October 2016 at 02:53, Amit Langote <Langote_Amit_f8@lab.ntt.co.jp> wrote:
>>>
>>> Please find attached the latest version of the patches
>>
>> Hi,
>>
>> I started to review the functionality of this patch, so i applied all
>> 9 patches. After that i found this warning, which i guess is because
>> it needs a cast.
>>
> 
> oh! i forgot the warning
> """
> partition.c: In function ‘get_qual_for_list’:
> partition.c:1159:6: warning: assignment from incompatible pointer type
>    or = makeBoolExpr(OR_EXPR, list_make2(nulltest2, opexpr), -1);
>       ^
> """

This one I noticed too and have fixed.

> 
> attached a list of the warnings that my compiler give me (basically
> most are just variables that could be used uninitialized)

Thanks a lot for spotting and reporting these.  Will fix as appropriate.

Regards,
Amit

Hi Jaime,

On 2016/11/08 2:15, Jaime Casanova wrote:
> On 28 October 2016 at 02:53, Amit Langote <Langote_Amit_f8@lab.ntt.co.jp> wrote:
> 
> I started to review the functionality of this patch, so i applied all
> 9 patches. After that i found this warning, which i guess is because
> it needs a cast.

Thanks a ton for reviewing!

> After that, i tried a case that i think is important: to partition an
> already existing table. Because there is no ALTER TABL SET PARTITION
> or something similar (which i think makes sense because such a command
> would need to create the partitions and move the rows to follow the
> rule that there is no rows in a parent table).
> 
> So, what i tried was:
> 
> 1) rename original table
> 2) create a new partitioned table with old name
> 3) attach old table as a partition with bounds outside normal bounds
> and no validate
> 
> the idea is to start attaching valid partitions and delete and insert
> rows from the invalid one (is there a better way of doing that?), that
> will allow to partition a table easily.

So, step 3 creates a partition that is basically unbounded.  From there,
it seems you want to create partitions with proper bounds, moving data
into them as they are created.  It seems like a job of some redistribution
command (split partition?) which is currently unsupported.

> So far so good, until i decided points 1 to 3 should happen inside a
> transaction to make things transparent to the user.
> 
> Attached is script that shows the failure when trying it:
> 
> script 1 (failing_test_1.sql) fails the assert
> "Assert(RelationGetPartitionKey(parentRel) != NULL);" in
> transformAttachPartition() at src/backend/parser/parse_utilcmd.c:3164

Thanks for the test.  This test uncovered a bug which I have fixed in my
local repository.  Given the fix the above will work, although I see that
it's not the best way to do what you want to do.

> After that i tried the same but with an already partitioned (via
> inheritance) table and got this (i did this first without a
> transaction, doing it with a transaction will show the same failure as
> before):
> 
> script 2 (failing_test_2.sql) fails the assert
> "Assert(childrte->relkind == RELKIND_PARTITIONED_TABLE);" in
> expand_inherited_rte_internal() at
> src/backend/optimizer/prep/prepunion.c:1551

This again was an oversight/bug in the patch.  It's not supported to
combine old-style inheritance partitioning with the new partitioned
tables.  In fact, ATTACH PARTITION prevents adding a regular inheritance
parent as partition.  After fixing the bug, you would instead get this error:

alter table prueba attach partition prueba_old for values start
(unbounded, unbounded) end (2008,1) no validate;
ERROR:  cannot attach regular inheritance parent as partition

In this case, you should instead create prueba_old partition hierarchy
using the new partitioning commands and then attach the same.

> PS: i don't like the START END syntax but i don't have any ideas
> there... except, there was a reason not to use expressions (like a
> CHECK constraint?)

Expression syntax is too unrestricted.  What's to prevent users from using
completely different check constraint expressions from partition to
partition (not that any users deliberately try do that)?  With the new
partitioning syntax, you specify the partitioning columns once when
creating the partitioned parent and then specify, using partitioning
method specific syntax, the bounds for every partition of the parent.
That allows to capture the partition metadata in a form that is more
suitable to implement other features related to partitioning.

That said, I think it's always a challenge to come up with a syntax that
is universally acceptable.  For example, the recent discussion about
whether to allow inclusive/exclusive to be specified for START and END
bounds of range partitions or always assume inclusive start and exclusive
end [1].

Thanks,
Amit

[1]
https://www.postgresql.org/message-id/CA%2BTgmoaKOycHcVoed%2BF3fk-z6xUOeysQFG6HT%3Doucw76bSMHCQ%40mail.gmail.com

On 2016/11/04 9:16, Robert Haas wrote:
> Apologies if I've made some of these comments before and/or missed
> comments you've made on these topics.  The size of this patch set is
> so large that it's hard to keep track of everything.

Thanks for the reviews!

> Re-reviewing 0001:
>
> +       indicate which table columns are used as partition key.  For example,
>
> s/are used as/are part of the/

Fixed.

>
> +       third table columns make up the partition key.  A zero in this array
> +       indicates that the corresponding partition key column is an expression
> +       over the table columns, rather than a simple column reference.
>
> I think you can leave out "over the table columns".

Done.

> +      columns or expressions forms the <firstterm>partitioning key</firstterm>
>
> s/forms/form/

Actually the sentence there is: The parenthesized list of columns or
expressions forms the the <firstterm>partitioning key</firstterm> for the
table

>
> +      The table itself is empty.  A data row inserted into the table is routed
>
> s/The table/The partitioned table/

Done.

>
> +     * Anything mentioned in the expressions.  We must ignore the column
> +     * references which will count as self-dependency items; in this case,
> +     * the depender is the table itself (there is no such thing as partition
> +     * key object).
>
> "depender" is not really a word, and the parenthetical note isn't very
> clear.  Maybe: We must ignore the column references, which will depend
> on the table itself; there is no separate partition key object.

I see "depender" being used as a variable name, but I guess it's not
appropriate to use the same in comments.  In any case, I adopted your text.

>
> +    heap_close(pg_partitioned_table, RowExclusiveLock);
>
> It seems like it would be safe to do this right after
> CatalogUpdateIndexes(pg_partitioned_table, tuple), and I'd recommend
> that you do.  Not for performance or anything, but just to keep
> related code together.

I see, done.

>
>  /*
>   * Resolve possibly-defaulted operator class specification
>   */
> -static Oid
> +Oid
>  GetIndexOpClass(List *opclass, Oid attrType,
>
> Perhaps we should rename this function to ResolveOpClass, since it's
> now going to be used for both indexes and partitioning keys.

Aha, good idea!  Done.

> + * Sets *is_expr if attnum is found to be referenced in some partition key
> + * expression.
>
> is_expr doesn't seem quite as clear as, say, used_by_expr or used_in_expr.
>
> Also, the specification for this function doesn't seem to be very
> clear about what this is supposed to do if the same column is both an
> explicit partitioning column and also used in an expression, and the
> code looks like it'll return with *is_expr set based on whichever use
> it encounters first.  If that's intended behavior, maybe add a comment
> like: It's possible for a column to be used both directly and as part
> of a partition key expression; if that happens, *is_expr may end up as
> either true or false.  That's OK for current uses of this function,
> because *is_expr is only used to tailor the error message text.

OK, I added the note as you suggest.

>
> +            if (is_expr)
> +                *is_expr = false;
> +            if (attnum == partattno)
> +                return true;
>
> I think you should adjust this (and the other bit in the same
> function) so that you don't set *is_expr until you're committed to
> returning.

Done.

>
> +            index = -1;
> +            while ((index = bms_next_member(expr_attrs, index)) > 0)
> +            {
> +                AttrNumber attno = index + FirstLowInvalidHeapAttributeNumber;
> +
> +                if (attno == attnum)
> +                    return true;
> +            }
>
> How about bms_is_member(expr_attrs, attnum -
> FirstLowInvalidHeapAttributeNumber), instead of looping?

Done that way, thanks.

> +                 errmsg("cannot reference relation \"%s\"",
> RelationGetRelationName(pkrel)),
> +                 errdetail("Referencing partitioned tables in foreign
> key constraints is not supported.")));
>
> I think you could get rid of the errdetail and just have the error
> message be "cannot reference partitioned table \"%s\"".

OK, done.

>
> +                         errmsg("column \"%s\" appears twice in
> partition key", pelem->name),
>
> It could be there three times!  How about column \"%s\" appears more
> than once in partition key?  (I see that you seem to have adapted this
> from some code in parse_utilcmd.c, which perhaps should also be
> adjusted, but that's a job for another patch.)

Done.

>
> +            /*
> +             * Strip any top-level COLLATE clause.  This ensures that we treat
> +             * "x COLLATE y" and "(x COLLATE y)" alike.
> +             */
>
> But you don't, right?  Unless I am confused, which is possible, the
> latter COLLATE will be ignored, while the former one will set the
> collation to be used in the context of partitioning comparisons.

The code immediately following the comment does in fact strip the
top-level COLLATE clause.

            while (IsA(expr, CollateExpr))
                expr = (Node *) ((CollateExpr *) expr)->arg;

So that the following two specifications are equivalent which is the intent:

create table p (a text) partition by range (a collate "en_US");
vs.
create table p (a text) partition by range ((a collate "en_US"));

> Re-reviewing 0002:
>
> +       if (fout->remoteVersion >= 100000)
> +       {
> +               PQExpBuffer acl_subquery = createPQExpBuffer();
> +               PQExpBuffer racl_subquery = createPQExpBuffer();
> +               PQExpBuffer initacl_subquery = createPQExpBuffer();
> +               PQExpBuffer initracl_subquery = createPQExpBuffer();
> +
> +               PQExpBuffer attacl_subquery = createPQExpBuffer();
> +               PQExpBuffer attracl_subquery = createPQExpBuffer();
> +               PQExpBuffer attinitacl_subquery = createPQExpBuffer();
> +               PQExpBuffer attinitracl_subquery = createPQExpBuffer();
>
> It seems unnecessary to repeat all of this.  The only differences
> between the 10000 version and the 9600 version are:
>
> 60,61c60
> <                           "AS changed_acl, "
> <                           "CASE WHEN c.relkind = 'P' THEN
> pg_catalog.pg_get_partkeydef(c.oid) ELSE NULL END AS partkeydef "
> ---
>>                           "AS changed_acl "
> 73c72
> <                    "WHERE c.relkind in ('%c', '%c', '%c', '%c',
> '%c', '%c', '%c') "
> ---
>>                    "WHERE c.relkind in ('%c', '%c', '%c', '%c', '%c', '%c') "
> 87,88c86
> <                           RELKIND_MATVIEW, RELKIND_FOREIGN_TABLE,
> <                           RELKIND_PARTITIONED_TABLE);
> ---
>>                           RELKIND_MATVIEW, RELKIND_FOREIGN_TABLE);
>
> But none of that is really a problem.  Sure, the 'P' case will never
> arise in 9.6, but so what?  I'd really like to not keep duplicating
> these increasingly-complex hunks of code if we can find some way to
> avoid that.

We cannot reference pg_catalog.pg_get_partkeydef() in the SQL query that
getTables() sends to pre-10 servers, right?  But I suppose we need not
call it in that particular SQL query in the first place.

How about we do it in the following manner in getSchemaData():

if (g_verbose)
    write_msg(NULL, "reading partition key information for interesting
tables\n");
getTablePartitionKeyInfo(fout, tblinfo, numTables);

I have implemented the same.

>
>         /* find all the inheritance information */
>
> -       appendPQExpBufferStr(query, "SELECT inhrelid, inhparent FROM
> pg_inherits");
> +       appendPQExpBufferStr(query,
> +                                                "SELECT inhrelid, inhparent "
> +                                                "FROM pg_inherits "
> +                                                "WHERE inhparent NOT
> IN (SELECT oid FROM pg_class WHERE relkind = 'P')");
>
> I think you need to update the comment.  "Find inheritance
> information, excluding implicit inheritance via partitioning.  We're
> not interested in that case because $REASON."

I just realized that this hunk really belongs in patch 0004.  In any case,
I added the explanatory comment like you suggest.

> Re-reviewing 0003:
>
> +     <para>
> +      If this table is a partition, one cannot perform <literal>DROP
> NOT NULL</>
> +      on a column if it is marked not null in the parent table.
> +     </para>
>
> This would, presumably, also be true for inheritance.  I think we
> could just leave this out.

Actually, it isn't true for the regular inheritance situation.

create table parent (a int not null);
create table child () inherits (parent);
alter table child alter a drop not null;  -- this works (bug?)

vs.

create table p (a int, b int, c int) partition by range (a);
create table p1 partition of p for values start (1) end (10);
alter table p1 alter a drop not null;  -- this causes error


Once NOT NULL constraints start using pg_constraint [1] to keep track of
inheritance (which currently is not kept track of), the above illustrated
bug will be fixed.  If that patch gets committed, it will have taken care
of the partitioning case as well.

But in the meantime, we can proceed with enforcing inheritance on NOT NULL
constraint for *partitions*, because they only ever have one parent and
hence do not need elaborate coninhcount based scheme, I think.  Thoughts?

>
> +      as partition of the target table.  The partition bound specification must
>
> s/as partition/as a partition/

Fixed.

>
> +      correspond to the partitioning method and partitioning key of the target
>
> I think that in most places were are referring to the "partitioning
> method" (with ing) but the "partition key" (without ing). Let's try to
> be consistent.

I'm inclined to switch to "partitioning method" and "partitioning key",
but do you mean just the documentation or throughout?  Beside
documentation, I mean source code comments, error messages, etc.  I have
assumed throughout.

>
> +      table.  The table to be attached must have all the columns as the target
> +      table and no more; moreover, the column types must also match.  Also, it
> +      must have all the matching constraints as the target table.
>
> s/all the columns/all of the same columns/

Fixed.

> The second sentence doesn't seem quite grammatical.  And why would
> that be true anyway?  Partitions can have extra constraints, and if
> they lack constraints that are present on the partitioned table, those
> constraints will be added and validated, right?

It's true that partitions can have extra constraints which don't affect
the discussion.  What I meant to say with the second sentence is that any
check constraints defined on the parent table (being referred to as the
target table) must be present in the table being attached.  It's the same
rule as for regular inheritance.  I didn't change how that works in the
partitioning case.  So no, check constraints of the parent table that are
missing in the table being attached are not added and validated as part of
the attach partition command processing.

I did change the second sentence to: Also, it must have all the
<literal>NOT NULL</literal> and <literal>CHECK</literal> constraints
present in the target table.

>
> +      A full table scan is performed on the table being attached to check that
> +      existing rows in the table are within the specified partition bound.
> +      If it is known in advance that no partition bound violating rows are
> +      present in the table, the above scan can be skipped by specifying the
> +      <literal>NO VALIDATE</> option.
> +      Default behavior is to perform the scan, as if the
> <literal>VALIDATE</literal>
> +      option were specified.
>
> I don't think it's OK to allow the partition to be added if it
> contains rows that might not be valid.  We are generally vary wary
> about adding options that can cause data integrity failures and I
> think we shouldn't start here, either.  On the other hand, it's also
> not desirable for adding a partition to take O(n) time in all cases.
> So what would be nice is if the new partition could self-certify that
> contains no problematic rows by having a constraint that matches the
> new partitioning constraint.  Then we can skip the scan if that
> constraint is already present.

I agree that NO VALIDATE is undesirable and it would be better if we could
get rid of the same.  I want to clarify one thing though: what does it
mean for the new partition to have a constraint that *matches* the new
partitioning constraint?  Does it mean the new partition's constraint
*implies* the partitioning constraint?  Or as certified by equal()?

>
> +      inherited columns.  One can also specify table constraints, in addition
>
> Delete comma.

Done.

>
> +      to those inherited from the parent.  If a check constraint with the name
> +      matching one of the parent's constraint is specified, it is merged with
> +      the latter, provided the specified condition is same.
>
> Doesn't that effectively delete the merged constraint?
>
> Suggest: "If the parent already has a check constraint with the same
> name as a constraint specified for the child, the conditions must be
> the same."
>
> +) FOR VALUES IN ('los angeles', 'san fransisco');
>
> That's not you you spell San Francisco.

Oops.  I changed the example such that specified values are properly
capitalized.

<programlisting>
CREATE TABLE cities (
    name         text not null,
    population   int,
) PARTITION BY LIST (initcap(name));
</programlisting></para>

<programlisting>
CREATE TABLE cities_west
    PARTITION OF cities (
    CONSTRAINT city_id_nonzero CHECK (city_id != 0)
) FOR VALUES IN ('Los Angeles', 'San Francisco');
</programlisting></para>

Previously, the key was: PARTITION BY LIST (lower(name))

> +   Create partition of a list partitioned table that itself is further
> +   partitioned and then create its partition:
>
> s/itself is/is itself/
> s/then create its partition/then add a partition to it/

Fixed.

>
> +                               if (!is_local || con->coninhcount == 0)
> +                                       con->coninhcount++;
>
> I would think that you could get rid of the "if" and just say
> con->coninhcount = 1.

That's better.  So:

/*
 * In case of partitions, an inherited constraint must be
 * inherited only once since it cannot have multiple parents and
 * it is never considered local.
 */
if (rel->rd_rel->relispartition)
{
    con->coninhcount = 1;
    con->conislocal = false;
}

The above is enforced in both MergeWithExistingConstraint() and
MergeConstraintsIntoExisting().

> It seems to me (and perhaps the comment could
> say this) that for a partitioned table, we can simplify the handling
> of coninhcount and conislocal.  Since partitioning hierarchies don't
> allow multiple inheritance, any inherited constraint must be inherited
> exactly once.  Since a partition cannot have inheritance children --
> except by being partitioned itself -- there is no point in tracking
> conislocal, so we always store it as false.

Agreed.

> +
> +void
> +StorePartitionBound(Relation rel, Node *bound)
>
> Header comment, please!

Sorry about that, added.

>
> +       (void) SysCacheGetAttr(RELOID, tuple, Anum_pg_class_relpartbound,
> +                                                  &isnull);
> +       Assert(isnull);
>
> We try not to do unnecessary work in non-assert-enabled builds solely
> for the benefit of assert-enabled builds.  We also try not to end up
> with variables that are only used in assert-enabled builds but not
> marked PG_USED_FOR_ASSERTS_ONLY, because those tend to cause compiler
> warnings.  I'm not sure an compiler would be smart enough to warn
> about this, but I suggest adding an #ifdef USE_ASSERT_CHECKING with a
> block inside where the offending variable is declared.  Actually, I
> think you need to move a bit more code.  Hmm.  Something like:
>
> #ifdef USE_ASSERT_CHECKING
>     {
>         Form_pg_class classForm;
>         bool isnull;
>
>        classForm = (Form_pg_class) GETSTRUCT(tuple);
>        Assert(!classForm->relispartition);
>
>        (void) SysCacheGetAttr(RELOID, tuple, Anum_pg_class_relpartbound,
>                                                   &isnull);
>        Assert(isnull);
>     }
> #endif

Done this way, thanks.

>
> + * are same in common cases, of which we only store one.
>
> "and we only store one of them."

Done.

>
> +                                                       /*
> +                                                        * Never put a
> null into the values array, flag
> +                                                        * instead for
> the code further down below where
> +                                                        * we
> construct the actual relcache struct.
> +                                                        */
> +
> found_null_partition = true;
> +
> null_partition_index = i;
>
> How about we elog(ERROR, ...) if found_null_partition is already set?

Makes sense.  However, let me mention here that duplicates either within
one partition's list or across the partitions are not possible.  That's
because in case of the former, we de-duplicate before storing the list
into the catalog and the latter would simply be an overlap error.  Could
this be made an Assert() then?

>
> +                               foreach(cell, non_null_values)
> +                               {
> +                                       PartitionListValue      *src =
> lfirst(cell);
> +
> +                                       all_values[i] = (PartitionListValue *)
> +
>          palloc(sizeof(PartitionListValue));
> +                                       all_values[i]->value =
> datumCopy(src->value,
> +
>                                   key->parttypbyval[0],
> +
>                                   key->parttyplen[0]);
> +                                       all_values[i]->index = src->index;
> +                                       i++;
> +                               }
>
> Why do we need to datumCopy() here when we just did that in the
> previous loop?  Why did the previous loop need to datumCopy(), anyway?
>  Can't we just pass the same datums around?  I understand that you
> need to copy the data into the correct context at the end, but doing
> two copies prior to that seems excessive.

Agreed. Now the datumCopying() happens only once when copying to the
relcache context.

> +get_leaf_partition_oids(Oid relid, int lockmode)
>
> Instead of implementing this, can you just use find_all_inheritors()?

OK, got rid of get_leaf_partition_oids().

>
> +               /*
> +                * Is lower_val = upper_val?
> +                */
> +               if (lower_val && upper_val)
>
> So, that comment does not actually match that code.  That if-test is
> just checking whether both bounds are finite.  What I think you should
> be explaining here is that if lower_val and upper_val are both
> non-infinite, and if the happen to be equal, we want to emit an
> equality test rather than a >= test plus a <= test because $REASON.

OK, I have added some explanatory comments around this code.

>
> +       operoid = get_opfamily_member(key->partopfamily[col],
> +
> key->parttypid[col],
> +
> key->parttypid[col],
> +                                                                 strategy);
> +       if (!OidIsValid(operoid))
> +       {
> +               operoid = get_opfamily_member(key->partopfamily[col],
> +
>    key->partopcintype[col],
> +
>    key->partopcintype[col],
> +
>    strategy);
> +               *need_relabel = true;
> +       }
>
> There's a comment explaining THAT you do this ("Use either the column
> type as the operator datatype or opclass's declared input type.") but,
> to repeat a complaint that I've made often, nothing explaining why.
> In this particular case, what's not clear - in my opinion - is why you
> need to try two different possibilities, and why at least one of those
> possibilities is guaranteed to work.  I gather that if the opfamily
> doesn't contain an operator for the actual type of the partitioning
> column, you think it will certainly contain one for the input type of
> the operator class (which seems right), and that the input type of the
> operator class will be binary-compatible with the type of the
> partitioning column (which is less-obviously true, and needs some
> explanation).

Sorry, it is indeed not clear why the code is the way it it.  I expanded
the comment explaining why (borrowing from what you wrote above).

>
> I also think that this function should elog(ERROR, ...) if by any
> chance the second get_opfamily_member() call also fails. Otherwise the
> error might occur quite a bit downstream and be hard to understand.

OK, added an elog(ERROR, ...) after the second get_opfamily_member() call.

>
> +       ReleaseSysCache(tuple);
> +       heap_close(parent, AccessShareLock);
>
> I think you had better not release the lock here - i.e. pass NoLock.
> We don't release locks on tables until transaction commit, except for
> catalog tables.  This also comes up in, at least,
> transformPartitionOf().

Ah, fixed.

> +    PartitionRangeBound *b1 = (*(PartitionRangeBound *const *) a);
> +    PartitionRangeBound *b2 = (*(PartitionRangeBound *const *) b);
> +       PartitionKey key = (PartitionKey) arg;
> +
> +       return partition_rbound_cmp(key, b1, b2);
>
> Whitespace.

Fixed.

>
> +        * as partition and schema consists of columns definitions corresponding
>
> the schema consists
>
> -               if (recurse)
> +               /* Force inheritance recursion, if partitioned table. */
> +               if (recurse || rel->rd_rel->relkind ==
> RELKIND_PARTITIONED_TABLE)
>
> I would instead error out if ONLY is specified for a partitioned table.

I forgot this instance of forced-recursion-if-partitioned, fixed.  I was
not quite sure what the error message would say; how about:

ereport(ERROR,
        (errcode(ERRCODE_WRONG_OBJECT_TYPE),
         errmsg("must truncate child tables too")));

> +               /*
> +                * If the table is source table of ATTACH PARTITION
> command, following
> +                * check is unnecessary.
> +                */
>
> As usual, comment should say WHY.

OK, expanded the comment explaining why.

>
> +                       if (partqualstate && !ExecQual(partqualstate,
> econtext, true))
> +                               ereport(ERROR,
> +
> (errcode(ERRCODE_CHECK_VIOLATION),
> +                                                errmsg("child table
> contains a row violating partition bound specification")));
>
> Why not mimic the existing phrasing?  "partition constraint is
> violated by some row"

Agreed, done.

>
> What happens if you try to attach a table as a partition of itself or
> one of its ancestors?

The latter fails with "already a partition" error.

The former case was not being handled at all which has now been fixed.
ATExecAddInherit() prevents that case as an instance of preventing the
circularity of inheritance.  It says: ERROR: circular inheritance not
allowed.  And then: DETAIL: "rel" is already a child of "rel".

Should ATExecAttachPartition() use the same trick and keep using the same
message(s)?  The above detail message doesn't quite sound appropriate when
one tries to attach a table as partition of itself.

>
> -                               errmsg("column \"%s\" in child table
> must be marked NOT NULL",
> -                                          attributeName)));
> +                                                errmsg("column \"%s\"
> in child table must be marked NOT NULL",
> +
> attributeName)));
>
> Whitespace-only hunk; revert.  You cannot fight the power of pgindent.

Oops, fixed.

>
> +                                errmsg("cannot attach table that is a
> inheritance child as partition")));
>
> an inheritance child
>
> +               errmsg("cannot attach a temporary relation of another
> session as partition ")));
>
> Extra space.

Fixed.

>
> +                                        errdetail("Table being
> attached should contain only the columns"
> +                                                          " present
> in parent.")));
>
> Suggest: "New partition should contain only..."
>
> Also, don't break error messages into multiple strings.  Make it one
> long string and let pgindent deal.

Done.

> +                       | FOR VALUES START '(' range_datum_list ')' lb_inc
> +                                                END_P '('
> range_datum_list ')' ub_inc
>
> Just a random idea.  Would for VALUES FROM ( ... ) TO ( ... ) be more
> idiomatic than START and END?

It would actually.  Should I go ahead and change to FROM (...) TO (...)?

Related to range partitioning, should we finalize on inclusive START/FROM
and exclusive END/TO preventing explicit specification of the inclusivity?

> +static void
> +transformPartitionOf(CreateStmtContext *cxt, Node *bound)
> +{
> +       TupleDesc       tupdesc;
> +       int                     i;
> +       RangeVar   *part = cxt->relation;
> +       RangeVar   *partof = linitial(cxt->inhRelations);
> +       Relation        parentRel;
> +
> +       parentRel = heap_openrv(partof, AccessShareLock);
> +
>
> If there's anyway that we might do heap_openrv() on the same RangeVar
> at multiple places in the code, it presents security and integrity
> hazards because the referent of that RangeVar might change in the
> meantime.  I suspect there is such a risk here - won't we need to open
> the relation again when we actually want to do the operation?
>
> Also, we should never acquire a lower-level lock on a relation and
> then, further downstream, acquire a higher-level lock on the same
> object.  To do so creates a deadlock hazard.  That seems like it might
> be a problem here, too.

I think I have modified things so that the concerns you express are taken
care of.  In particular, I got rid of transformPartitionOf() altogether,
moving part of its responsibilities to MergeAttributes() where the parent
table is locked anyway and calling transformPartitionBound from
DefineRelation() (we needed access to the parent's PartitionKey for that).

>
> +                               /*if (ldatum->infinite && rdatum->infinite)
> +                                       ereport(ERROR,
> +
> (errcode(ERRCODE_INVALID_TABLE_DEFINITION),
> +                                                errmsg("both START
> and END datum for column \"%s\" cannot be UNBOUNDED",
> +                                                               colname),
> +
> parser_errposition(cxt->pstate, rdatum->location)));*/
>
> Commented-out code is bad.

Oops, removed.

>
> +                       if (list_length(spec->lowerdatums) > partnatts)
> +                               ereport(ERROR,
> +
> (errcode(ERRCODE_INVALID_TABLE_DEFINITION),
> +                                        errmsg("START has more values
> than the length of the partition key"),
> +
> parser_errposition(cxt->pstate,
> +
> exprLocation(list_nth(spec->lowerdatums,
> +
>   list_length(spec->lowerdatums) - 1)))));
> +                       else if (list_length(spec->lowerdatums) < partnatts)
> +                               ereport(ERROR,
> +
> (errcode(ERRCODE_INVALID_TABLE_DEFINITION),
> +                                        errmsg("START has fewer
> values than the length of the partition key"),
> +
> parser_errposition(cxt->pstate,
> +
> exprLocation(list_nth(spec->lowerdatums,
> +
>   list_length(spec->lowerdatums) - 1)))));
>
> It would be worth looking for a way to unify these cases.  Like "START
> must specify exactly one value per partitioning column".

Agreed.  The code is trying to generate too specific error messages.

> +                * Same oids? No mind order - in the list case, it
> matches the order
> +                * in which partition oids are returned by a
> pg_inherits scan, whereas
> +                * in the range case, they are in order of ranges of individual
> +                * partitions.  XXX - is the former unsafe?
>
> Probably.  It likely depends on the physical ordering of tuples in the
> table, which can change.

This comment is outdated.  RelationBuildPartitionDesc() now always puts
the OIDs into the array in a canonical order (in both list and range
cases) using.  I have updated the comment.

>
> + * BoundCollection encapsulates a set of partition bounds of either physical
> + * or logical relations.  It is associated with a partitioned relation of
> + * which the aforementioned relations are partitions.
>
> "physical or logical relations" is unfamiliar terminology.

What I am trying to say there is that one can associate a BoundCollection
with either an actual partitioned table or with a transient partitioned
relation, say, a partitioned joinrel.  In case of the former, the
BoundCollection is part of the table's partition descriptor.  I can see
though that "logical/physical relations" terminology is misleading at
best.  I rewrote the comment:

/*
 * BoundCollection encapsulates a set of partition bounds.  It is usually
 * associated with partitioned tables as part of its partition descriptor.
 *
 * The internal structure is opaque outside partition.c.
 */
typedef struct BoundCollectionData *BoundCollection;


Attached updated patches take care of the above comments and few other
fixes.  There are still a few items I have not addressed right away:

- Remove NO VALIDATE clause in ATTACH PARTITION and instead rely on the
  new partition's constraints to skip the validation scan
- Remove the syntax to specify inclusivity of each of START and END bounds
  of range partitions and instead assume inclusive START and exclusive END

Also, what used to be "[PATCH 5/9] Refactor optimizer's inheritance set
expansion code" is no longer included in the set.  I think we can go back
to that topic later as I mentioned last week [2].

Thanks,
Amit

[1] https://www.postgresql.org/message-id/16589.1465997664%40sss.pgh.pa.us
[2]
https://www.postgresql.org/message-id/0a299fbf-e5a1-dc3d-eb5e-11d3601eac16%40lab.ntt.co.jp

In this latest patch set:

src/backend/parser/parse_utilcmd.c:3194: indent with spaces.
+                                    *rdatum;

With all patches applied, "make check" fails with a bunch of diffs
that look like this:
 Check constraints:
-     "pt1chk2" CHECK (c2 <> ''::text)     "pt1chk3" CHECK (c2 <> ''::text)


On Wed, Nov 9, 2016 at 6:14 AM, Amit Langote
<Langote_Amit_f8@lab.ntt.co.jp> wrote:
> Actually the sentence there is: The parenthesized list of columns or
> expressions forms the the <firstterm>partitioning key</firstterm> for the
> table

OK.

>> +            /*
>> +             * Strip any top-level COLLATE clause.  This ensures that we treat
>> +             * "x COLLATE y" and "(x COLLATE y)" alike.
>> +             */
>>
>> But you don't, right?  Unless I am confused, which is possible, the
>> latter COLLATE will be ignored, while the former one will set the
>> collation to be used in the context of partitioning comparisons.
>
> The code immediately following the comment does in fact strip the
> top-level COLLATE clause.
>
>             while (IsA(expr, CollateExpr))
>                 expr = (Node *) ((CollateExpr *) expr)->arg;
>
> So that the following two specifications are equivalent which is the intent:
>
> create table p (a text) partition by range (a collate "en_US");
> vs.
> create table p (a text) partition by range ((a collate "en_US"));

I see.  You're right.

>> Re-reviewing 0002:
>>
>> +       if (fout->remoteVersion >= 100000)
>> +       {
>> +               PQExpBuffer acl_subquery = createPQExpBuffer();
>> +               PQExpBuffer racl_subquery = createPQExpBuffer();
>> +               PQExpBuffer initacl_subquery = createPQExpBuffer();
>> +               PQExpBuffer initracl_subquery = createPQExpBuffer();
>> +
>> +               PQExpBuffer attacl_subquery = createPQExpBuffer();
>> +               PQExpBuffer attracl_subquery = createPQExpBuffer();
>> +               PQExpBuffer attinitacl_subquery = createPQExpBuffer();
>> +               PQExpBuffer attinitracl_subquery = createPQExpBuffer();
>>
>> It seems unnecessary to repeat all of this.  The only differences
>> between the 10000 version and the 9600 version are:
>>
>> 60,61c60
>> <                           "AS changed_acl, "
>> <                           "CASE WHEN c.relkind = 'P' THEN
>> pg_catalog.pg_get_partkeydef(c.oid) ELSE NULL END AS partkeydef "
>> ---
>>>                           "AS changed_acl "
>> 73c72
>> <                    "WHERE c.relkind in ('%c', '%c', '%c', '%c',
>> '%c', '%c', '%c') "
>> ---
>>>                    "WHERE c.relkind in ('%c', '%c', '%c', '%c', '%c', '%c') "
>> 87,88c86
>> <                           RELKIND_MATVIEW, RELKIND_FOREIGN_TABLE,
>> <                           RELKIND_PARTITIONED_TABLE);
>> ---
>>>                           RELKIND_MATVIEW, RELKIND_FOREIGN_TABLE);
>>
>> But none of that is really a problem.  Sure, the 'P' case will never
>> arise in 9.6, but so what?  I'd really like to not keep duplicating
>> these increasingly-complex hunks of code if we can find some way to
>> avoid that.
>
> We cannot reference pg_catalog.pg_get_partkeydef() in the SQL query that
> getTables() sends to pre-10 servers, right?  But I suppose we need not
> call it in that particular SQL query in the first place.

Oh, yeah, that's a problem; the query will error out against older
servers.  You could do something like:

char *partkeydef;
if (version <= 90600)   partkeydef = "NULL";
else   partkeydef = "CASE WHEN c.relkind = 'P' THEN
pg_catalog.pg_get_partkeydef(c.oid) ELSE NULL END";

...and the use %s to interpolate that into the query string.

> How about we do it in the following manner in getSchemaData():
>
> if (g_verbose)
>     write_msg(NULL, "reading partition key information for interesting
> tables\n");
> getTablePartitionKeyInfo(fout, tblinfo, numTables);
>
> I have implemented the same.

That might be OK too; I'll have to read it through carefully.

>> Re-reviewing 0003:
>>
>> +     <para>
>> +      If this table is a partition, one cannot perform <literal>DROP
>> NOT NULL</>
>> +      on a column if it is marked not null in the parent table.
>> +     </para>
>>
>> This would, presumably, also be true for inheritance.  I think we
>> could just leave this out.
>
> Actually, it isn't true for the regular inheritance situation.
>
> create table parent (a int not null);
> create table child () inherits (parent);
> alter table child alter a drop not null;  -- this works (bug?)

Hrm, OK.  That doesn't satisfy MY idea of the principle of least
astonishment, but hey...

> But in the meantime, we can proceed with enforcing inheritance on NOT NULL
> constraint for *partitions*, because they only ever have one parent and
> hence do not need elaborate coninhcount based scheme, I think.  Thoughts?

I agree.

>> +      correspond to the partitioning method and partitioning key of the target
>>
>> I think that in most places were are referring to the "partitioning
>> method" (with ing) but the "partition key" (without ing). Let's try to
>> be consistent.
>
> I'm inclined to switch to "partitioning method" and "partitioning key",
> but do you mean just the documentation or throughout?  Beside
> documentation, I mean source code comments, error messages, etc.  I have
> assumed throughout.

I think "partitioning key" is a bit awkward and actually prefer
"partiton key".  But "partition method" sounds funny so I would go
with "partitioning method".

>> +      A full table scan is performed on the table being attached to check that
>> +      existing rows in the table are within the specified partition bound.
>> +      If it is known in advance that no partition bound violating rows are
>> +      present in the table, the above scan can be skipped by specifying the
>> +      <literal>NO VALIDATE</> option.
>> +      Default behavior is to perform the scan, as if the
>> <literal>VALIDATE</literal>
>> +      option were specified.
>>
>> I don't think it's OK to allow the partition to be added if it
>> contains rows that might not be valid.  We are generally vary wary
>> about adding options that can cause data integrity failures and I
>> think we shouldn't start here, either.  On the other hand, it's also
>> not desirable for adding a partition to take O(n) time in all cases.
>> So what would be nice is if the new partition could self-certify that
>> contains no problematic rows by having a constraint that matches the
>> new partitioning constraint.  Then we can skip the scan if that
>> constraint is already present.
>
> I agree that NO VALIDATE is undesirable and it would be better if we could
> get rid of the same.  I want to clarify one thing though: what does it
> mean for the new partition to have a constraint that *matches* the new
> partitioning constraint?  Does it mean the new partition's constraint
> *implies* the partitioning constraint?  Or as certified by equal()?

Implies would be better, but equal() might be tolerable.

>> +                                                       /*
>> +                                                        * Never put a
>> null into the values array, flag
>> +                                                        * instead for
>> the code further down below where
>> +                                                        * we
>> construct the actual relcache struct.
>> +                                                        */
>> +
>> found_null_partition = true;
>> +
>> null_partition_index = i;
>>
>> How about we elog(ERROR, ...) if found_null_partition is already set?
>
> Makes sense.  However, let me mention here that duplicates either within
> one partition's list or across the partitions are not possible.  That's
> because in case of the former, we de-duplicate before storing the list
> into the catalog and the latter would simply be an overlap error.  Could
> this be made an Assert() then?

Well, I think that wouldn't be as good, because for example some
misguided person might do a manual update of the catalog.  It seems to
me that if the system catalog contents are questionable, it's better
to error out than to have completely undefined behavior.  In other
words, we probably want it checked in non-Assert builds.  See similar
cases where we have things like:

elog(ERROR, "conpfeqop is not a 1-D Oid array");

>> +               /*
>> +                * Is lower_val = upper_val?
>> +                */
>> +               if (lower_val && upper_val)
>>
>> So, that comment does not actually match that code.  That if-test is
>> just checking whether both bounds are finite.  What I think you should
>> be explaining here is that if lower_val and upper_val are both
>> non-infinite, and if the happen to be equal, we want to emit an
>> equality test rather than a >= test plus a <= test because $REASON.
>
> OK, I have added some explanatory comments around this code.

So, I was kind of assuming that the answer to "why are we doing this"
was "for efficiency".  It's true that val >= const AND const <= val
can be simplified to val = const, but it wouldn't be necessary to do
so for correctness.  It just looks nicer and runs faster.

By the way, if you want to keep the inclusive stuff, I think you
probably need to consider this logic in light of the possibility that
one bound might be exclusive.  Maybe you're thinking that can't happen
anyway because it would have been rejected earlier, but an elog(ERROR,
...) might still be appropriate just to guard against messed-up
catalogs.

> I forgot this instance of forced-recursion-if-partitioned, fixed.  I was
> not quite sure what the error message would say; how about:
>
> ereport(ERROR,
>         (errcode(ERRCODE_WRONG_OBJECT_TYPE),
>          errmsg("must truncate child tables too")));

Works fror me.

>> What happens if you try to attach a table as a partition of itself or
>> one of its ancestors?
>
> The latter fails with "already a partition" error.
>
> The former case was not being handled at all which has now been fixed.
> ATExecAddInherit() prevents that case as an instance of preventing the
> circularity of inheritance.  It says: ERROR: circular inheritance not
> allowed.  And then: DETAIL: "rel" is already a child of "rel".
>
> Should ATExecAttachPartition() use the same trick and keep using the same
> message(s)?  The above detail message doesn't quite sound appropriate when
> one tries to attach a table as partition of itself.

Whichever way is easier to code seems fine.  It's a pretty obscure
case, so I don't think we need to add code just to get a very slightly
better error message.

>> +                       | FOR VALUES START '(' range_datum_list ')' lb_inc
>> +                                                END_P '('
>> range_datum_list ')' ub_inc
>>
>> Just a random idea.  Would for VALUES FROM ( ... ) TO ( ... ) be more
>> idiomatic than START and END?
>
> It would actually.  Should I go ahead and change to FROM (...) TO (...)?

+1!

> Related to range partitioning, should we finalize on inclusive START/FROM
> and exclusive END/TO preventing explicit specification of the inclusivity?

I would be in favor of committing the initial patch set without that,
and then considering the possibility of adding it later.  If we
include it in the initial patch set we are stuck with it.

> Attached updated patches take care of the above comments and few other
> fixes.  There are still a few items I have not addressed right away:
>
> - Remove NO VALIDATE clause in ATTACH PARTITION and instead rely on the
>   new partition's constraints to skip the validation scan
> - Remove the syntax to specify inclusivity of each of START and END bounds
>   of range partitions and instead assume inclusive START and exclusive END

OK - what is the time frame for these changes?

> Also, what used to be "[PATCH 5/9] Refactor optimizer's inheritance set
> expansion code" is no longer included in the set.  I think we can go back
> to that topic later as I mentioned last week [2].

Great.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Wed, Nov 9, 2016 at 12:00 PM, Robert Haas <robertmhaas@gmail.com> wrote:
> In this latest patch set:
>
> src/backend/parser/parse_utilcmd.c:3194: indent with spaces.
> +                                    *rdatum;
>
> With all patches applied, "make check" fails with a bunch of diffs
> that look like this:
>
>   Check constraints:
> -     "pt1chk2" CHECK (c2 <> ''::text)
>       "pt1chk3" CHECK (c2 <> ''::text)

And the pg_upgrade test also fails:

Done
+ pg_dumpall -f /Users/rhaas/pgsql/src/bin/pg_upgrade/tmp_check/dump2.sql
+ pg_ctl -m fast stop
waiting for server to shut down.... done
server stopped
+ set +x

Files /Users/rhaas/pgsql/src/bin/pg_upgrade/tmp_check/dump1.sql and
/Users/rhaas/pgsql/src/bin/pg_upgrade/tmp_check/dump2.sql differ
dumps were not identical
make[2]: *** [check] Error 1
make[1]: *** [check-pg_upgrade-recurse] Error 2
make: *** [check-world-src/bin-recurse] Error 2
[rhaas pgsql]$ diff
/Users/rhaas/pgsql/src/bin/pg_upgrade/tmp_check/dump1.sql
/Users/rhaas/pgsql/src/bin/pg_upgrade/tmp_check/dump2.sql
6403d6402
<     c text
8736,8737c8735
<     CONSTRAINT blocal CHECK (((b)::double precision < (1000)::double
precision)),
<     CONSTRAINT bmerged CHECK (((b)::double precision > (1)::double precision))
---
>     CONSTRAINT blocal CHECK (((b)::double precision < (1000)::double precision))

For future revisions, please make sure "make check-world" passes before posting.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 2016/11/10 2:00, Robert Haas wrote:
> In this latest patch set:
> 
> src/backend/parser/parse_utilcmd.c:3194: indent with spaces.
> +                                    *rdatum;

This one I will fix.

> 
> With all patches applied, "make check" fails with a bunch of diffs
> that look like this:
> 
>   Check constraints:
> -     "pt1chk2" CHECK (c2 <> ''::text)
>       "pt1chk3" CHECK (c2 <> ''::text)

Hm, I can't seem to reproduce this one.  Is it perhaps possible that you
applied the patches on top of some other WIP patches or something?

> And the pg_upgrade test also fails:
>
> Done
> + pg_dumpall -f /Users/rhaas/pgsql/src/bin/pg_upgrade/tmp_check/dump2.sql
> + pg_ctl -m fast stop
> waiting for server to shut down.... done
> server stopped
> + set +x
>
> Files /Users/rhaas/pgsql/src/bin/pg_upgrade/tmp_check/dump1.sql and
> /Users/rhaas/pgsql/src/bin/pg_upgrade/tmp_check/dump2.sql differ
> dumps were not identical
> make[2]: *** [check] Error 1
> make[1]: *** [check-pg_upgrade-recurse] Error 2
> make: *** [check-world-src/bin-recurse] Error 2
> [rhaas pgsql]$ diff
> /Users/rhaas/pgsql/src/bin/pg_upgrade/tmp_check/dump1.sql
> /Users/rhaas/pgsql/src/bin/pg_upgrade/tmp_check/dump2.sql
> 6403d6402
> <     c text
> 8736,8737c8735
> <     CONSTRAINT blocal CHECK (((b)::double precision < (1000)::double
> precision)),
> <     CONSTRAINT bmerged CHECK (((b)::double precision > (1)::double
precision))
> ---
> >     CONSTRAINT blocal CHECK (((b)::double precision < (1000)::double
precision))

This one too I can't seem to reproduce.

> For future revisions, please make sure "make check-world" passes before
posting.

OK, I will make sure.  FWIW, make check-world passes here after applying
the patches posted yesterday.

Thanks,
Amit

Thanks again for the prompt review!

On 2016/11/10 2:00, Robert Haas wrote:
> On Wed, Nov 9, 2016 at 6:14 AM, Amit Langote
> <Langote_Amit_f8@lab.ntt.co.jp> wrote:
>>> But none of that is really a problem.  Sure, the 'P' case will never
>>> arise in 9.6, but so what?  I'd really like to not keep duplicating
>>> these increasingly-complex hunks of code if we can find some way to
>>> avoid that.
>>
>> We cannot reference pg_catalog.pg_get_partkeydef() in the SQL query that
>> getTables() sends to pre-10 servers, right?  But I suppose we need not
>> call it in that particular SQL query in the first place.
>
> Oh, yeah, that's a problem; the query will error out against older
> servers.  You could do something like:
>
> char *partkeydef;
> if (version <= 90600)
>     partkeydef = "NULL";
> else
>     partkeydef = "CASE WHEN c.relkind = 'P' THEN
> pg_catalog.pg_get_partkeydef(c.oid) ELSE NULL END";
>
> ...and the use %s to interpolate that into the query string.

Yeah, that's a way.

>>> I think that in most places were are referring to the "partitioning
>>> method" (with ing) but the "partition key" (without ing). Let's try to
>>> be consistent.
>>
>> I'm inclined to switch to "partitioning method" and "partitioning key",
>> but do you mean just the documentation or throughout?  Beside
>> documentation, I mean source code comments, error messages, etc.  I have
>> assumed throughout.
>
> I think "partitioning key" is a bit awkward and actually prefer
> "partiton key".  But "partition method" sounds funny so I would go
> with "partitioning method".

OK, "partition key" and "partitioning method" it is then.  Source code
comments, error messages, variables call the latter (partitioning)
"strategy" though which hopefully is fine.

>>> I don't think it's OK to allow the partition to be added if it
>>> contains rows that might not be valid.  We are generally vary wary
>>> about adding options that can cause data integrity failures and I
>>> think we shouldn't start here, either.  On the other hand, it's also
>>> not desirable for adding a partition to take O(n) time in all cases.
>>> So what would be nice is if the new partition could self-certify that
>>> contains no problematic rows by having a constraint that matches the
>>> new partitioning constraint.  Then we can skip the scan if that
>>> constraint is already present.
>>
>> I agree that NO VALIDATE is undesirable and it would be better if we could
>> get rid of the same.  I want to clarify one thing though: what does it
>> mean for the new partition to have a constraint that *matches* the new
>> partitioning constraint?  Does it mean the new partition's constraint
>> *implies* the partitioning constraint?  Or as certified by equal()?
>
> Implies would be better, but equal() might be tolerable.

I think a equal() -based method would fail to help in most cases. Consider
the following example:

create table foo1 (a int);
alter table foo add constraint check_a check (a < 10 and a >= 1);

create table foo (a int) partition by range (a);
alter table foo attach partition foo1 for values from (1) to (10);

The last command will internally generate the partition constraints that
is basically a list of implicitly AND'd expressions viz. a is not null, a
>= 1 and a < 10 which we wrap into a BoolExpr.  It would not be equal()
with foo1's constraint even though they are basically the same constraint.
 So, simple structural equality may prove to be less productive, IMHO.

It seems a *implies* -based solution would work much better, although a
bit slower for obvious reasons.  I reckon slownsess is not a big issue in
this case.  So I prototyped the same using predicate_implied_by() which
seems to work reasonably.  Looks something like this:

    skip_validate = false;
    if (predicate_implied_by(partConstraint, existConstraint))
        skip_validate = true;

Where partConstraint is 1-member list with the new partition constraint
and existConstraint is a list of the existing constraints of the table
being attached, derived from its TupleConstr.

>>> +                                                       /*
>>> +                                                        * Never put a
>>> null into the values array, flag
>>> +                                                        * instead for
>>> the code further down below where
>>> +                                                        * we
>>> construct the actual relcache struct.
>>> +                                                        */
>>> +
>>> found_null_partition = true;
>>> +
>>> null_partition_index = i;
>>>
>>> How about we elog(ERROR, ...) if found_null_partition is already set?
>>
>> Makes sense.  However, let me mention here that duplicates either within
>> one partition's list or across the partitions are not possible.  That's
>> because in case of the former, we de-duplicate before storing the list
>> into the catalog and the latter would simply be an overlap error.  Could
>> this be made an Assert() then?
>
> Well, I think that wouldn't be as good, because for example some
> misguided person might do a manual update of the catalog.  It seems to
> me that if the system catalog contents are questionable, it's better
> to error out than to have completely undefined behavior.  In other
> words, we probably want it checked in non-Assert builds.  See similar
> cases where we have things like:
>
> elog(ERROR, "conpfeqop is not a 1-D Oid array");

Good point, agreed.

>>> +               /*
>>> +                * Is lower_val = upper_val?
>>> +                */
>>> +               if (lower_val && upper_val)
>>>
>>> So, that comment does not actually match that code.  That if-test is
>>> just checking whether both bounds are finite.  What I think you should
>>> be explaining here is that if lower_val and upper_val are both
>>> non-infinite, and if the happen to be equal, we want to emit an
>>> equality test rather than a >= test plus a <= test because $REASON.
>>
>> OK, I have added some explanatory comments around this code.
>
> So, I was kind of assuming that the answer to "why are we doing this"
> was "for efficiency".  It's true that val >= const AND const <= val
> can be simplified to val = const, but it wouldn't be necessary to do
> so for correctness.  It just looks nicer and runs faster.

I guess I ended up with this code following along a bit different way of
thinking about it, but in the end what you're saying is true.

> By the way, if you want to keep the inclusive stuff, I think you
> probably need to consider this logic in light of the possibility that
> one bound might be exclusive.  Maybe you're thinking that can't happen
> anyway because it would have been rejected earlier, but an elog(ERROR,
> ...) might still be appropriate just to guard against messed-up
> catalogs.

Yes, that makes sense.  I guess you mean a case like the one shown below:

create table foo5 partition of foo for values start (3, 10) end (3, 10);
ERROR:  cannot create range partition with empty range

I think the following would suffice as a guard (checked only if it turns
out that lower_val and upper_val are indeed equal):

    if (i == key->partnatts - 1 && spec->lowerinc != spec->upperinc)
        elog(ERROR, "invalid range bound specification");


>>> What happens if you try to attach a table as a partition of itself or
>>> one of its ancestors?
>>
>> The latter fails with "already a partition" error.
>>
>> The former case was not being handled at all which has now been fixed.
>> ATExecAddInherit() prevents that case as an instance of preventing the
>> circularity of inheritance.  It says: ERROR: circular inheritance not
>> allowed.  And then: DETAIL: "rel" is already a child of "rel".
>>
>> Should ATExecAttachPartition() use the same trick and keep using the same
>> message(s)?  The above detail message doesn't quite sound appropriate when
>> one tries to attach a table as partition of itself.
>
> Whichever way is easier to code seems fine.  It's a pretty obscure
> case, so I don't think we need to add code just to get a very slightly
> better error message.

OK, so let's keep it same as with regular inheritance.

>>> +                       | FOR VALUES START '(' range_datum_list ')' lb_inc
>>> +                                                END_P '('
>>> range_datum_list ')' ub_inc
>>>
>>> Just a random idea.  Would for VALUES FROM ( ... ) TO ( ... ) be more
>>> idiomatic than START and END?
>>
>> It would actually.  Should I go ahead and change to FROM (...) TO (...)?
>
> +1!

Done!

>> Related to range partitioning, should we finalize on inclusive START/FROM
>> and exclusive END/TO preventing explicit specification of the inclusivity?
>
> I would be in favor of committing the initial patch set without that,
> and then considering the possibility of adding it later.  If we
> include it in the initial patch set we are stuck with it.

OK, I have removed the syntactic ability to specify INCLUSIVE/EXCLUSIVE
with each of the range bounds.

I haven't changed any code (such as comparison functions) that manipulates
instances of PartitionRangeBound which has a flag called inclusive.  I
didn't remove the flag, but is instead just set to (is_lower ? true :
false) when initializing from the parse node. Perhaps, there is some scope
for further simplifying that code, which you probably alluded to when you
proposed that we do this.

>> Attached updated patches take care of the above comments and few other
>> fixes.  There are still a few items I have not addressed right away:
>>
>> - Remove NO VALIDATE clause in ATTACH PARTITION and instead rely on the
>>   new partition's constraints to skip the validation scan
>> - Remove the syntax to specify inclusivity of each of START and END bounds
>>   of range partitions and instead assume inclusive START and exclusive END
>
> OK - what is the time frame for these changes?

I have implemented both of these in the attached patch.  As mentioned
above, the logic to skip the validation scan using the new partition's
constraints is still kind of a prototype solution, but it seems to work so
far.  Comments on the same would be very helpful.

Thanks,
Amit

On Thu, Nov 10, 2016 at 7:40 AM, Amit Langote
<Langote_Amit_f8@lab.ntt.co.jp> wrote:
>> I think "partitioning key" is a bit awkward and actually prefer
>> "partiton key".  But "partition method" sounds funny so I would go
>> with "partitioning method".
>
> OK, "partition key" and "partitioning method" it is then.  Source code
> comments, error messages, variables call the latter (partitioning)
> "strategy" though which hopefully is fine.

Oh, I like "partitioning strategy".  Can we standardize on that?

>>> Related to range partitioning, should we finalize on inclusive START/FROM
>>> and exclusive END/TO preventing explicit specification of the inclusivity?
>>
>> I would be in favor of committing the initial patch set without that,
>> and then considering the possibility of adding it later.  If we
>> include it in the initial patch set we are stuck with it.
>
> OK, I have removed the syntactic ability to specify INCLUSIVE/EXCLUSIVE
> with each of the range bounds.
>
> I haven't changed any code (such as comparison functions) that manipulates
> instances of PartitionRangeBound which has a flag called inclusive.  I
> didn't remove the flag, but is instead just set to (is_lower ? true :
> false) when initializing from the parse node. Perhaps, there is some scope
> for further simplifying that code, which you probably alluded to when you
> proposed that we do this.

Yes, you need to rip out all of the logic that supports it.  Having
the logic to support it but not the syntax is bad because then that
code can't be properly tested.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Wed, Nov 9, 2016 at 9:58 PM, Amit Langote
<Langote_Amit_f8@lab.ntt.co.jp> wrote:
>> With all patches applied, "make check" fails with a bunch of diffs
>> that look like this:
>>
>>   Check constraints:
>> -     "pt1chk2" CHECK (c2 <> ''::text)
>>       "pt1chk3" CHECK (c2 <> ''::text)
>
> Hm, I can't seem to reproduce this one.  Is it perhaps possible that you
> applied the patches on top of some other WIP patches or something?

Nope.  I just checked and this passes with only 0001 and 0002 applied,
but when I add 0003 and 0004 then it starts failing.  It appears that
the problem starts at this point in the foreign_data test:

ALTER TABLE pt1 DROP CONSTRAINT pt1chk2 CASCADE;

After that command, in the expected output, pt1chk2 stops showing up
in the output of \d+ pt1, but continues to appear in the output of \d+
ft2.  With your patch, however, it stops showing up for ft2 also.  If
that's not also happening for you, it might be due to an uninitialized
variable someplace.

+        /* Force inheritance recursion, if partitioned table. */

Doesn't match code (any more).

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 2016/11/11 6:51, Robert Haas wrote:
> On Wed, Nov 9, 2016 at 9:58 PM, Amit Langote
> <Langote_Amit_f8@lab.ntt.co.jp> wrote:
>>> With all patches applied, "make check" fails with a bunch of diffs
>>> that look like this:
>>>
>>>   Check constraints:
>>> -     "pt1chk2" CHECK (c2 <> ''::text)
>>>       "pt1chk3" CHECK (c2 <> ''::text)
>>
>> Hm, I can't seem to reproduce this one.  Is it perhaps possible that you
>> applied the patches on top of some other WIP patches or something?
>
> Nope.  I just checked and this passes with only 0001 and 0002 applied,
> but when I add 0003 and 0004 then it starts failing.

Sorry, it definitely wasn't an error on your part.

> It appears that
> the problem starts at this point in the foreign_data test:
>
> ALTER TABLE pt1 DROP CONSTRAINT pt1chk2 CASCADE;
>
> After that command, in the expected output, pt1chk2 stops showing up
> in the output of \d+ pt1, but continues to appear in the output of \d+
> ft2.  With your patch, however, it stops showing up for ft2 also.  If
> that's not also happening for you, it might be due to an uninitialized
> variable someplace.

Thanks for the context.  I think I found the culprit variable in
MergeConstraintsIntoExisting() and fixed it.  As you correctly guessed,
the uninitialized variable caused (in your environment) even non-partition
child relations to be treated partitions and hence forced any merged
constraints to be non-local in all cases, not just in case of partitions.
Which meant the command you quoted would even drop the ft2's (a child)
constraint because its conislocal is wrongly false.

>
> +        /* Force inheritance recursion, if partitioned table. */
>
> Doesn't match code (any more).

Fixed.

>>> I think "partitioning key" is a bit awkward and actually prefer
>>> "partiton key".  But "partition method" sounds funny so I would go
>>> with "partitioning method".
>>
>> OK, "partition key" and "partitioning method" it is then.  Source code
>> comments, error messages, variables call the latter (partitioning)
>> "strategy" though which hopefully is fine.
>
> Oh, I like "partitioning strategy".  Can we standardize on that?

OK, done.

>>> I would be in favor of committing the initial patch set without that,
>>> and then considering the possibility of adding it later.  If we
>>> include it in the initial patch set we are stuck with it.
>>
>> OK, I have removed the syntactic ability to specify INCLUSIVE/EXCLUSIVE
>> with each of the range bounds.
>>
>> I haven't changed any code (such as comparison functions) that manipulates
>> instances of PartitionRangeBound which has a flag called inclusive.  I
>> didn't remove the flag, but is instead just set to (is_lower ? true :
>> false) when initializing from the parse node. Perhaps, there is some scope
>> for further simplifying that code, which you probably alluded to when you
>> proposed that we do this.
>
> Yes, you need to rip out all of the logic that supports it.  Having
> the logic to support it but not the syntax is bad because then that
> code can't be properly tested.

Agreed, done.


Attached updated patches.

Thanks,
Amit

I have not looked at the latest set of patches, but in the version
that I have we create one composite type for every partition. This
means that if there are thousand partitions, there will be thousand
identical entries in pg_type. Since all the partitions share the same
definition (by syntax), it doesn't make sense to add so many identical
entries. Moreover, in set_append_rel_size(), while translating the
expressions from parent to child, we add a ConvertRowtypeExpr instead
of whole-row reference if reltype of the parent and child do not match
(adjust_appendrel_attrs_mutator())                  if (appinfo->parent_reltype != appinfo->child_reltype)
    {                       ConvertRowtypeExpr *r = makeNode(ConvertRowtypeExpr);
 

I guess, we should set reltype of child to that of parent for
declarative partitions.

On Fri, Nov 11, 2016 at 4:00 PM, Amit Langote
<Langote_Amit_f8@lab.ntt.co.jp> wrote:
> On 2016/11/11 6:51, Robert Haas wrote:
>> On Wed, Nov 9, 2016 at 9:58 PM, Amit Langote
>> <Langote_Amit_f8@lab.ntt.co.jp> wrote:
>>>> With all patches applied, "make check" fails with a bunch of diffs
>>>> that look like this:
>>>>
>>>>   Check constraints:
>>>> -     "pt1chk2" CHECK (c2 <> ''::text)
>>>>       "pt1chk3" CHECK (c2 <> ''::text)
>>>
>>> Hm, I can't seem to reproduce this one.  Is it perhaps possible that you
>>> applied the patches on top of some other WIP patches or something?
>>
>> Nope.  I just checked and this passes with only 0001 and 0002 applied,
>> but when I add 0003 and 0004 then it starts failing.
>
> Sorry, it definitely wasn't an error on your part.
>
>> It appears that
>> the problem starts at this point in the foreign_data test:
>>
>> ALTER TABLE pt1 DROP CONSTRAINT pt1chk2 CASCADE;
>>
>> After that command, in the expected output, pt1chk2 stops showing up
>> in the output of \d+ pt1, but continues to appear in the output of \d+
>> ft2.  With your patch, however, it stops showing up for ft2 also.  If
>> that's not also happening for you, it might be due to an uninitialized
>> variable someplace.
>
> Thanks for the context.  I think I found the culprit variable in
> MergeConstraintsIntoExisting() and fixed it.  As you correctly guessed,
> the uninitialized variable caused (in your environment) even non-partition
> child relations to be treated partitions and hence forced any merged
> constraints to be non-local in all cases, not just in case of partitions.
> Which meant the command you quoted would even drop the ft2's (a child)
> constraint because its conislocal is wrongly false.
>
>>
>> +        /* Force inheritance recursion, if partitioned table. */
>>
>> Doesn't match code (any more).
>
> Fixed.
>
>>>> I think "partitioning key" is a bit awkward and actually prefer
>>>> "partiton key".  But "partition method" sounds funny so I would go
>>>> with "partitioning method".
>>>
>>> OK, "partition key" and "partitioning method" it is then.  Source code
>>> comments, error messages, variables call the latter (partitioning)
>>> "strategy" though which hopefully is fine.
>>
>> Oh, I like "partitioning strategy".  Can we standardize on that?
>
> OK, done.
>
>>>> I would be in favor of committing the initial patch set without that,
>>>> and then considering the possibility of adding it later.  If we
>>>> include it in the initial patch set we are stuck with it.
>>>
>>> OK, I have removed the syntactic ability to specify INCLUSIVE/EXCLUSIVE
>>> with each of the range bounds.
>>>
>>> I haven't changed any code (such as comparison functions) that manipulates
>>> instances of PartitionRangeBound which has a flag called inclusive.  I
>>> didn't remove the flag, but is instead just set to (is_lower ? true :
>>> false) when initializing from the parse node. Perhaps, there is some scope
>>> for further simplifying that code, which you probably alluded to when you
>>> proposed that we do this.
>>
>> Yes, you need to rip out all of the logic that supports it.  Having
>> the logic to support it but not the syntax is bad because then that
>> code can't be properly tested.
>
> Agreed, done.
>
>
> Attached updated patches.
>
> Thanks,
> Amit



-- 
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company

I forgot to quote your comments in the email I sent on Friday [1], with
new patches that do take care of the following comments.

On 2016/11/11 4:04, Robert Haas wrote:
> On Thu, Nov 10, 2016 at 7:40 AM, Amit Langote
>>
>> OK, "partition key" and "partitioning method" it is then.  Source code
>> comments, error messages, variables call the latter (partitioning)
>> "strategy" though which hopefully is fine.
> 
> Oh, I like "partitioning strategy".  Can we standardize on that?

Done.

>> OK, I have removed the syntactic ability to specify INCLUSIVE/EXCLUSIVE
>> with each of the range bounds.
>>
>> I haven't changed any code (such as comparison functions) that manipulates
>> instances of PartitionRangeBound which has a flag called inclusive.  I
>> didn't remove the flag, but is instead just set to (is_lower ? true :
>> false) when initializing from the parse node. Perhaps, there is some scope
>> for further simplifying that code, which you probably alluded to when you
>> proposed that we do this.
> 
> Yes, you need to rip out all of the logic that supports it.  Having
> the logic to support it but not the syntax is bad because then that
> code can't be properly tested.

Agreed and done.  Now the only thing that dictates the inclusivity of
individual range bounds during comparisons (with other range bounds or
partition key of tuples) is whether the bound is a lower bound or not;
inclusive if, exclusive if not.

Thanks,
Amit

[1]
https://www.postgresql.org/message-id/8d7c35e3-1c85-33d0-4440-0a75bf9d31cd%40lab.ntt.co.jp

On 2016/11/04 0:49, Robert Haas wrote:
> On Thu, Nov 3, 2016 at 7:46 AM,  <alvherre@alvh.no-ip.org> wrote:
>> El 2016-10-28 07:53, Amit Langote escribió:
>>> @@ -6267,6 +6416,12 @@ ATAddForeignKeyConstraint(AlteredTableInfo *tab,
>>> Relation rel,
>>>          * Validity checks (permission checks wait till we have the column
>>>          * numbers)
>>>          */
>>> +       if (pkrel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
>>> +               ereport(ERROR,
>>> +                               (errcode(ERRCODE_WRONG_OBJECT_TYPE),
>>> +                                errmsg("cannot reference relation
>>> \"%s\"", RelationGetRelationName(pkrel)),
>>> +                                errdetail("Referencing partitioned tables
>>> in foreign key constraints is not supported.")));
>>
>> Is there a plan for fixing this particular limitation?  It's a pretty
>> serious problem for users,
>> and the suggested workaround (to create a separate non-partitioned table
>> which carries only the PK
>> columns which is updated by triggers, and direct the FKs to it instead of to
>> the partitioned table)
>> is not only a very ugly one, but also very slow.
> 
> If you have two compatibly partitioned tables, and the foreign key
> matches the partitioning keys, you could implement a foreign key
> between the two tables as a foreign key between each pair of matching
> partitions.  Otherwise, isn't the only way to handle this a global
> index?

I am assuming you don't mean a global index (on partitioned tables) as in
some new kind of monolithic physical structure that implements the
constraint across tables (partitions), right?  I'm thinking you mean a
collection of btree indexes on individual partitions with the key of each
index matching the partition key of the parent, created internally as part
of the creation of the same index on the parent.  In fact, the said
indexes are created and maintained sort of like how inherited attributes,
constraints are.  That would require quite a bit of new infrastructure.
We did discuss about the possibility of such a feature being implemented
on top of declarative partitioning, but not in version 1 [1].

Thanks,
Amit

[1]
https://www.postgresql.org/message-id/CA%2BTgmoZZMfcf16YaHuhP1Vk%3Dj8PDFeHCvfj%2BFJQd%2BeFhs%2B7P8A%40mail.gmail.com

On 2016/11/11 20:49, Ashutosh Bapat wrote:
> I have not looked at the latest set of patches, but in the version
> that I have we create one composite type for every partition. This
> means that if there are thousand partitions, there will be thousand
> identical entries in pg_type. Since all the partitions share the same
> definition (by syntax), it doesn't make sense to add so many identical
> entries. Moreover, in set_append_rel_size(), while translating the
> expressions from parent to child, we add a ConvertRowtypeExpr instead
> of whole-row reference if reltype of the parent and child do not match
> (adjust_appendrel_attrs_mutator())
>                    if (appinfo->parent_reltype != appinfo->child_reltype)
>                     {
>                         ConvertRowtypeExpr *r = makeNode(ConvertRowtypeExpr);
> 
> I guess, we should set reltype of child to that of parent for
> declarative partitions.

Thanks for the suggestion.  I agree that partitions having the same
reltype as the parent will help a number of cases including the one you
mentioned, but I haven't yet considered how invasive such a change will be.

Thanks,
Amit

On 2016/11/11 19:30, Amit Langote wrote:
>
> Attached updated patches.

Here is the latest version of the patches with some fixes along with those
mentioned below (mostly in 0003):

- Fixed the logic to skip the attach partition validation scan such that
  it won't skip scanning a list partition *that doesn't accept NULLs* if
  the partition key column is not set NOT NULL (it similarly doesn't skip
  scanning a range partition if either of the partition key columns is not
  set NOT NULL, because a range partition key cannot contain NULLs at all)

- Added some more regression tests for ATTACH PARTITION command

- Some fixes to documentation and source code comments

Thanks,
Amit

On Tue, Nov 15, 2016 at 5:30 AM, Amit Langote
<Langote_Amit_f8@lab.ntt.co.jp> wrote:
> On 2016/11/11 19:30, Amit Langote wrote:
>>
>> Attached updated patches.
>
> Here is the latest version of the patches with some fixes along with those
> mentioned below (mostly in 0003):
>
> - Fixed the logic to skip the attach partition validation scan such that
>   it won't skip scanning a list partition *that doesn't accept NULLs* if
>   the partition key column is not set NOT NULL (it similarly doesn't skip
>   scanning a range partition if either of the partition key columns is not
>   set NOT NULL, because a range partition key cannot contain NULLs at all)
>
> - Added some more regression tests for ATTACH PARTITION command
>
> - Some fixes to documentation and source code comments

Have you done any performance testing on the tuple routing code?
Suppose we insert a million (or 10 million) tuples into an
unpartitioned table, a table with 10 partitions, a table with 100
partitions, a table with 1000 partitions, and a table that is
partitioned into 10 partitions each of which has 10 subpartitions.
Ideally, the partitioned cases would run almost as fast as the
unpartitioned case, but probably there will be some overhead.
However, it would be useful to know how much.  Also, it would be
useful to set up the same cases with inheritance using a PL/pgsql ON
INSERT trigger for tuple routing and compare.  Hopefully the tuple
routing code is far faster than a trigger, but we should make sure
that's the case and look for optimizations if not.  Also, it would be
useful to know how much slower the tuple-mapping-required case is than
the no-tuple-mapping-required case.

I think the comments in some of the later patches could use some work
yet.  For example, in 0007, FormPartitionKeyDatum()'s header comment
is largely uninformative, get_partition_for_tuple()'s header comment
doesn't explain what the return value means in the non-failure case,
and ExecFindPartition() doesn't have a header comment at all.

The number of places in these patches where you end up reopening a
hopefully-already-locked relation with NoLock (or sometimes with
AccessShareLock) is worrying to me.  I think that's a coding pattern
we should be seeking to avoid; every one of those is not only a hazard
(what if we reach that point in the code without a lock?) but a
possible performance issue (we have to look up the OID in the
backend-private hash table; and if you are passing AccessShareLock
then you might also hit the lock manager which could be slow or create
deadlock hazards).  It would be much better to pass the Relation
around rather than the OID whenever possible.

Also, in 0006:

- I doubt that PartitionTreeNodeData's header comment will survive
contact with pgindent.

In 0007:

- oid_is_foreign_table could/should do a syscache lookup instead of
opening the heap relation.  But actually I think you could drop it
altogether and use get_rel_relkind().

- The XXX in EndCopy will need to be fixed somehow.

- I suspect that many of the pfree's in this patch are pointless
because the contexts in which those allocations are performed will be
reset or deleted shortly afterwards anyway.  Only pfree data that
might otherwise live substantially longer than we want, because pfree
consumes some cycles.

- The code in make_modifytable() to swap out the rte_array for a fake
one looks like an unacceptable kludge.  I don't know offhand what a
better design would look like, but what you've got is really ugly.

- I don't understand how it can be right for get_partition_for_tuple()
to be emitting an error that says "range partition key contains null".
A defective partition key should be detected at partition creation
time, not in the middle of tuple routing.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 2016/11/16 4:21, Robert Haas wrote:
> On Tue, Nov 15, 2016 at 5:30 AM, Amit Langote
> <Langote_Amit_f8@lab.ntt.co.jp> wrote:
>> On 2016/11/11 19:30, Amit Langote wrote:
>>>
>>> Attached updated patches.
>
> Have you done any performance testing on the tuple routing code?
> Suppose we insert a million (or 10 million) tuples into an
> unpartitioned table, a table with 10 partitions, a table with 100
> partitions, a table with 1000 partitions, and a table that is
> partitioned into 10 partitions each of which has 10 subpartitions.
> Ideally, the partitioned cases would run almost as fast as the
> unpartitioned case, but probably there will be some overhead.
> However, it would be useful to know how much.  Also, it would be
> useful to set up the same cases with inheritance using a PL/pgsql ON
> INSERT trigger for tuple routing and compare.  Hopefully the tuple
> routing code is far faster than a trigger, but we should make sure
> that's the case and look for optimizations if not.  Also, it would be
> useful to know how much slower the tuple-mapping-required case is than
> the no-tuple-mapping-required case.

OK, I will share the performance results soon.

Meanwhile, here are updated patch that address most of the following comments.

> I think the comments in some of the later patches could use some work
> yet.  For example, in 0007, FormPartitionKeyDatum()'s header comment
> is largely uninformative, get_partition_for_tuple()'s header comment
> doesn't explain what the return value means in the non-failure case,
> and ExecFindPartition() doesn't have a header comment at all.

Sorry, I have tried to make these comments more informative in the attached.

> The number of places in these patches where you end up reopening a
> hopefully-already-locked relation with NoLock (or sometimes with
> AccessShareLock) is worrying to me.  I think that's a coding pattern
> we should be seeking to avoid; every one of those is not only a hazard
> (what if we reach that point in the code without a lock?) but a
> possible performance issue (we have to look up the OID in the
> backend-private hash table; and if you are passing AccessShareLock
> then you might also hit the lock manager which could be slow or create
> deadlock hazards).  It would be much better to pass the Relation
> around rather than the OID whenever possible.

I have removed most instances of heap_open() calls after realizing they
were in fact unnecessary (and as you warn possibly hazardous and
performance hogs).  Where it's inevitable that we perform heap_open(), I
have changed it so that the correct lockmode is passed.

>
> Also, in 0006:
>
> - I doubt that PartitionTreeNodeData's header comment will survive
> contact with pgindent.

Fixed by adding "/* ----" at the top of the comment.

> In 0007:
>
> - oid_is_foreign_table could/should do a syscache lookup instead of
> opening the heap relation.  But actually I think you could drop it
> altogether and use get_rel_relkind().

Done with get_rel_relkind().

> - The XXX in EndCopy will need to be fixed somehow.

Fixed.  Realized that doing that in EndCopy() is kind of pointless, I
moved that code to the end of CopyFrom(), right before a call to
FreeExecutorState.

> - I suspect that many of the pfree's in this patch are pointless
> because the contexts in which those allocations are performed will be
> reset or deleted shortly afterwards anyway.  Only pfree data that
> might otherwise live substantially longer than we want, because pfree
> consumes some cycles.

They do seem to be freeing unnecessarily, removed.

>
> - The code in make_modifytable() to swap out the rte_array for a fake
> one looks like an unacceptable kludge.  I don't know offhand what a
> better design would look like, but what you've got is really ugly.

Agree that it looks horrible.  The problem is we don't add partition
(child table) RTEs when planning an insert on the parent and FDW
partitions can't do without some planner handling - planForeignModify()
expects a valid PlannerInfo for deparsing target lists (basically, to be
able to use planner_rt_fetch()).

Any design beside this kludge would perhaps mean that we will be adding
valid partition RTEs at some earlier planning stage much like what
expand_inherited_rtentry() does during non-insert queries.  Perhaps, we
could have expand_inherited_rtentry() make an exception in partitioned
tables' case and create root->append_rel_list members even in the insert
case.  We could then go over the append_rel_list in make_modifytable() to
get the valid RT index of foreign child tables to pass to
PlanForeignModify().  Using append_rel_list for insert planning is perhaps
equally ugly though. Thoughts?

> - I don't understand how it can be right for get_partition_for_tuple()
> to be emitting an error that says "range partition key contains null".
> A defective partition key should be detected at partition creation
> time, not in the middle of tuple routing.

That error is emitted when the partition key of the *input tuple* is found
to contain NULLs.  Maybe we need to consider something like how btree
indexes treat NULLs - order NULLs either before or after all the non-NULL
values based on NULLS FIRST/LAST config.  Thoughts?

Thanks,
Amit

On Thu, Nov 17, 2016 at 6:27 AM, Amit Langote
<Langote_Amit_f8@lab.ntt.co.jp> wrote:
> OK, I will share the performance results soon.

Thanks.

>> Also, in 0006:
>>
>> - I doubt that PartitionTreeNodeData's header comment will survive
>> contact with pgindent.
>
> Fixed by adding "/* ----" at the top of the comment.

OK.  Hopefully you also tested that.  In general, with a patch set
this large, the more you can do to make it pgindent-clean, the better.
Ideally none of your code would get changed by pgindent, or at least
not the new files.  But note that you will probably have have to
update typedefs.list if you actually want to be able to run it without
having it mangle things that involve your new structs.

>> - The code in make_modifytable() to swap out the rte_array for a fake
>> one looks like an unacceptable kludge.  I don't know offhand what a
>> better design would look like, but what you've got is really ugly.
>
> Agree that it looks horrible.  The problem is we don't add partition
> (child table) RTEs when planning an insert on the parent and FDW
> partitions can't do without some planner handling - planForeignModify()
> expects a valid PlannerInfo for deparsing target lists (basically, to be
> able to use planner_rt_fetch()).
>
> Any design beside this kludge would perhaps mean that we will be adding
> valid partition RTEs at some earlier planning stage much like what
> expand_inherited_rtentry() does during non-insert queries.  Perhaps, we
> could have expand_inherited_rtentry() make an exception in partitioned
> tables' case and create root->append_rel_list members even in the insert
> case.  We could then go over the append_rel_list in make_modifytable() to
> get the valid RT index of foreign child tables to pass to
> PlanForeignModify().  Using append_rel_list for insert planning is perhaps
> equally ugly though. Thoughts?

If it's only needed for foreign tables, how about for v1 we just throw
an error and say errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("cannot route inserted tuples to a foreign table") for now.  We
can come back and fix it later.  Doing more inheritance expansion
early is probably not a good idea because it likely sucks for
performance, and that's especially unfortunate if it's only needed for
foreign tables.  Coming up with some new FDW API or some modification
to the existing one is probably better, but I don't really want to get
hung up on that right now.

>> - I don't understand how it can be right for get_partition_for_tuple()
>> to be emitting an error that says "range partition key contains null".
>> A defective partition key should be detected at partition creation
>> time, not in the middle of tuple routing.
>
> That error is emitted when the partition key of the *input tuple* is found
> to contain NULLs.  Maybe we need to consider something like how btree
> indexes treat NULLs - order NULLs either before or after all the non-NULL
> values based on NULLS FIRST/LAST config.  Thoughts?

Well, I'd say my confusion suggests that the error message needs to be
clearer about what the problem is.  I think this is basically a case
of there being no partition for the given row, so maybe just arrange
to use that same message here ("no partition of relation \"%s\" found
for row").

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Thu, Nov 17, 2016 at 6:27 AM, Amit Langote
<Langote_Amit_f8@lab.ntt.co.jp> wrote:
> Meanwhile, here are updated patch that address most of the following comments.

OK, I have re-reviewed 0005 and it looks basically fine to me, modulo
a few minor nitpicks. "This is called, *iff*" shouldn't have a comma
there, and I think the entire comment block that starts with "NOTE:
SQL specifies that a NULL" and ends with "it's unlikely that NULL
would result." should be changed to say something like /* As for
catalogued constraints, we treat a NULL result as a success, not a
failure. */ rather than duplicating an existing comment that doesn't
quite apply here.  Finally, ExecConstraints() contains a new if-block
whose sole contents are another if-block.  Perhaps if (this && that)
would be better.

Regarding 0006 and 0007, I think the PartitionTreeNodeData structure
you've chosen is awfully convoluted and probably not that efficient.
For example, get_partition_for_tuple() contains this loop:

+       prev = parent;
+       node = parent->downlink;
+       while (node != NULL)
+       {
+               if (node->index >= cur_idx)
+                       break;
+
+               prev = node;
+               node = node->next;
+       }

Well, it looks to me like that's an O(n) way to find the n'th
partition, which seems like a pretty bad idea in performance-critical
code, which this is.  I think this whole structure needs a pretty
heavy overhaul.  Here's a proposal:

1. Forget the idea of a tree.  Instead, let the total number of tables
in the partitioning hierarchy be N and let the number of those that
are partitioned be K.  Assign each partitioned table in the hierarchy
an index between 0 and K-1.  Make your top level data structure (in
lieu of PartitionTreeNodeData) be an array of K PartitionDispatch
objects, with the partitioning root in entry 0 and the rest in the
remaining entries.

2. Within each PartitionDispatch object, store (a) a pointer to a
PartitionDesc and (b) an array of integers of length equal to the
PartitionDesc's nparts value.  Each integer i, if non-negative, is the
final return value for get_partition_for_tuple.  If i == -1, tuple
routing fails.  If i < -1, we must next route using the subpartition
whose PartitionDesc is at index -(i+1).  Arrange for the array to be
in the same order the PartitionDesc's OID list.

3. Now get_partition_for_tuple looks something like this:

K = 0
loop:   pd = PartitionDispatch[K]   idx = list/range_partition_for_tuple(pd->partdesc, ...)   if (idx >= -1)
returnidx   K = -(idx + 1)
 

No recursion, minimal pointer chasing, no linked lists.  The whole
thing is basically trivial aside from the cost of
list/range_partition_for_tuple itself; optimizing that is a different
project.  I might have some details slightly off here, but hopefully
you can see what I'm going for: you want to keep the computation that
happens in get_partition_for_tuple() to an absolute minimum, and
instead set things up in advance so that getting the partition for a
tuple is FAST.  And you want the data structures that you are using in
that process to be very compact, hence arrays instead of linked lists.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 2016/11/18 1:43, Robert Haas wrote:
> On Thu, Nov 17, 2016 at 6:27 AM, Amit Langote wrote:
>> OK, I will share the performance results soon.
> 
> Thanks.
> 
>>> Also, in 0006:
>>>
>>> - I doubt that PartitionTreeNodeData's header comment will survive
>>> contact with pgindent.
>>
>> Fixed by adding "/* ----" at the top of the comment.
> 
> OK.  Hopefully you also tested that.  In general, with a patch set
> this large, the more you can do to make it pgindent-clean, the better.
> Ideally none of your code would get changed by pgindent, or at least
> not the new files.  But note that you will probably have have to
> update typedefs.list if you actually want to be able to run it without
> having it mangle things that involve your new structs.

OK, I was afraid that running pgindent might end up introducing unrelated
diffs in the patch, but I'm sure any code committed to HEAD would have
been through pgindent and my worry might be pointless after all.  Also,
thanks for the tip about the typedefs.list.  I will go try pgindent'ing
the patches.

>>> - The code in make_modifytable() to swap out the rte_array for a fake
>>> one looks like an unacceptable kludge.  I don't know offhand what a
>>> better design would look like, but what you've got is really ugly.
>>
>> Agree that it looks horrible.  The problem is we don't add partition
>> (child table) RTEs when planning an insert on the parent and FDW
>> partitions can't do without some planner handling - planForeignModify()
>> expects a valid PlannerInfo for deparsing target lists (basically, to be
>> able to use planner_rt_fetch()).
>>
>> Any design beside this kludge would perhaps mean that we will be adding
>> valid partition RTEs at some earlier planning stage much like what
>> expand_inherited_rtentry() does during non-insert queries.  Perhaps, we
>> could have expand_inherited_rtentry() make an exception in partitioned
>> tables' case and create root->append_rel_list members even in the insert
>> case.  We could then go over the append_rel_list in make_modifytable() to
>> get the valid RT index of foreign child tables to pass to
>> PlanForeignModify().  Using append_rel_list for insert planning is perhaps
>> equally ugly though. Thoughts?
> 
> If it's only needed for foreign tables, how about for v1 we just throw
> an error and say errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
> errmsg("cannot route inserted tuples to a foreign table") for now.  We
> can come back and fix it later.  Doing more inheritance expansion
> early is probably not a good idea because it likely sucks for
> performance, and that's especially unfortunate if it's only needed for
> foreign tables.  Coming up with some new FDW API or some modification
> to the existing one is probably better, but I don't really want to get
> hung up on that right now.

OK, I agree with the decision to make tuple-routing a unsupported feature
in the first cut as far as foreign partitions are concerned.  Once can
still insert data into partitions directly.

I am assuming that the error should be thrown all the way below
ExecInsert(), not in the planner.  Which means I should revert any changes
I've made to the planner in this patch.

>>> - I don't understand how it can be right for get_partition_for_tuple()
>>> to be emitting an error that says "range partition key contains null".
>>> A defective partition key should be detected at partition creation
>>> time, not in the middle of tuple routing.
>>
>> That error is emitted when the partition key of the *input tuple* is found
>> to contain NULLs.  Maybe we need to consider something like how btree
>> indexes treat NULLs - order NULLs either before or after all the non-NULL
>> values based on NULLS FIRST/LAST config.  Thoughts?
> 
> Well, I'd say my confusion suggests that the error message needs to be
> clearer about what the problem is.  I think this is basically a case
> of there being no partition for the given row, so maybe just arrange
> to use that same message here ("no partition of relation \"%s\" found
> for row").

The reason NULLs in an input row are caught and rejected (with the current
message) before control reaches range_partition_for_tuple() is because
it's not clear to me whether the range bound comparison logic in
partition_rbound_datum_cmp() should be prepared to handle NULLs and what
the results of comparisons should look like.  Currently, all it ever
expects to see in the input tuple's partition key is non-NULL datums.
Comparison proceeds as follows: if a range bound datum is a finite value,
we invoke the comparison proc or if it is infinite, we conclude that the
input tuple is > or < the bound in question based on whether the bound is
a lower or upper bound, respectively.

Or are you saying that get_tuple_for_partition() should simply return -1
(partition not found) in case of encountering a NULL in range partition
key to the caller instead of throwing error as is now?  If the user sees
the message and decides to create a new range partition that *will* accept
such a row, how do they decide what its boundaries should be?

Thanks,
Amit

On 2016/11/18 4:14, Robert Haas wrote:
> On Thu, Nov 17, 2016 at 6:27 AM, Amit Langote
> <Langote_Amit_f8@lab.ntt.co.jp> wrote:
>> Meanwhile, here are updated patch that address most of the following comments.
>
> OK, I have re-reviewed 0005 and it looks basically fine to me, modulo
> a few minor nitpicks. "This is called, *iff*" shouldn't have a comma
> there,

Fixed.

> and I think the entire comment block that starts with "NOTE:
> SQL specifies that a NULL" and ends with "it's unlikely that NULL
> would result." should be changed to say something like /* As for
> catalogued constraints, we treat a NULL result as a success, not a
> failure. */ rather than duplicating an existing comment that doesn't
> quite apply here.

Ah, you're right that the comment does not apply as it is.  I rewrote that
comment.

Oh but wait, that means I can insert rows with NULLs in the range
partition key if I choose to insert it directly into the partition,
whereas I have been thinking all this while that there could never be
NULLs in the partition key of a range partition.  What's more,
get_qual_for_partbound() (patch 0003) emits a IS NOT NULL constraint for
every partition key column in case of a range partition.  Is that
wrongheaded altogether?  (also see my reply to your earlier message about
NULLs in the range partition key)

>  Finally, ExecConstraints() contains a new if-block
> whose sole contents are another if-block.  Perhaps if (this && that)
> would be better.

Agreed, should have noticed that.

> Regarding 0006 and 0007, I think the PartitionTreeNodeData structure
> you've chosen is awfully convoluted and probably not that efficient.
> For example, get_partition_for_tuple() contains this loop:
>
> +       prev = parent;
> +       node = parent->downlink;
> +       while (node != NULL)
> +       {
> +               if (node->index >= cur_idx)
> +                       break;
> +
> +               prev = node;
> +               node = node->next;
> +       }
>
> Well, it looks to me like that's an O(n) way to find the n'th
> partition, which seems like a pretty bad idea in performance-critical
> code, which this is.  I think this whole structure needs a pretty
> heavy overhaul.  Here's a proposal:

Thanks for the idea below!

> 1. Forget the idea of a tree.  Instead, let the total number of tables
> in the partitioning hierarchy be N and let the number of those that
> are partitioned be K.  Assign each partitioned table in the hierarchy
> an index between 0 and K-1.  Make your top level data structure (in
> lieu of PartitionTreeNodeData) be an array of K PartitionDispatch
> objects, with the partitioning root in entry 0 and the rest in the
> remaining entries.
>
> 2. Within each PartitionDispatch object, store (a) a pointer to a
> PartitionDesc and (b) an array of integers of length equal to the
> PartitionDesc's nparts value.  Each integer i, if non-negative, is the
> final return value for get_partition_for_tuple.  If i == -1, tuple
> routing fails.  If i < -1, we must next route using the subpartition
> whose PartitionDesc is at index -(i+1).  Arrange for the array to be
> in the same order the PartitionDesc's OID list.
>
> 3. Now get_partition_for_tuple looks something like this:
>
> K = 0
> loop:
>     pd = PartitionDispatch[K]
>     idx = list/range_partition_for_tuple(pd->partdesc, ...)
>     if (idx >= -1)
>         return idx
>     K = -(idx + 1)
>
> No recursion, minimal pointer chasing, no linked lists.  The whole
> thing is basically trivial aside from the cost of
> list/range_partition_for_tuple itself; optimizing that is a different
> project.  I might have some details slightly off here, but hopefully
> you can see what I'm going for: you want to keep the computation that
> happens in get_partition_for_tuple() to an absolute minimum, and
> instead set things up in advance so that getting the partition for a
> tuple is FAST.  And you want the data structures that you are using in
> that process to be very compact, hence arrays instead of linked lists.

This sounds *much* better.  Here is a quick attempt at coding the design
you have outlined above in the attached latest set of patches.

PS: I haven't run the patches through pgindent yet.

Thanks,
Amit

On Thu, Nov 17, 2016 at 8:18 PM, Amit Langote
<Langote_Amit_f8@lab.ntt.co.jp> wrote:
> The reason NULLs in an input row are caught and rejected (with the current
> message) before control reaches range_partition_for_tuple() is because
> it's not clear to me whether the range bound comparison logic in
> partition_rbound_datum_cmp() should be prepared to handle NULLs and what
> the results of comparisons should look like.  Currently, all it ever
> expects to see in the input tuple's partition key is non-NULL datums.
> Comparison proceeds as follows: if a range bound datum is a finite value,
> we invoke the comparison proc or if it is infinite, we conclude that the
> input tuple is > or < the bound in question based on whether the bound is
> a lower or upper bound, respectively.
>
> Or are you saying that get_tuple_for_partition() should simply return -1
> (partition not found) in case of encountering a NULL in range partition
> key to the caller instead of throwing error as is now?  If the user sees
> the message and decides to create a new range partition that *will* accept
> such a row, how do they decide what its boundaries should be?

Well, I think the first thing you have to decide is whether range
partitioning is going to support NULL values in the partition key
columns at all.  If you want it to support that, then you've got to
decide how it's going to be specified in the SQL syntax.  I had the
impression that you were planning not to support that, in which case
you need to reject all such rows.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Fri, Nov 18, 2016 at 5:59 AM, Amit Langote
<Langote_Amit_f8@lab.ntt.co.jp> wrote:
> Oh but wait, that means I can insert rows with NULLs in the range
> partition key if I choose to insert it directly into the partition,
> whereas I have been thinking all this while that there could never be
> NULLs in the partition key of a range partition.  What's more,
> get_qual_for_partbound() (patch 0003) emits a IS NOT NULL constraint for
> every partition key column in case of a range partition.  Is that
> wrongheaded altogether?  (also see my reply to your earlier message about
> NULLs in the range partition key)

The easiest thing to do might be to just enforce that all of the
partition key columns have to be not-null when the range-partitioned
table is defined, and reject any attempt to DROP NOT NULL on them
later.  That's probably better that shoehorning it into the table
constraint.

> Thanks for the idea below!
>
>> 1. Forget the idea of a tree.  Instead, let the total number of tables
>> in the partitioning hierarchy be N and let the number of those that
>> are partitioned be K.  Assign each partitioned table in the hierarchy
>> an index between 0 and K-1.  Make your top level data structure (in
>> lieu of PartitionTreeNodeData) be an array of K PartitionDispatch
>> objects, with the partitioning root in entry 0 and the rest in the
>> remaining entries.
>>
>> 2. Within each PartitionDispatch object, store (a) a pointer to a
>> PartitionDesc and (b) an array of integers of length equal to the
>> PartitionDesc's nparts value.  Each integer i, if non-negative, is the
>> final return value for get_partition_for_tuple.  If i == -1, tuple
>> routing fails.  If i < -1, we must next route using the subpartition
>> whose PartitionDesc is at index -(i+1).  Arrange for the array to be
>> in the same order the PartitionDesc's OID list.
>>
>> 3. Now get_partition_for_tuple looks something like this:
>>
>> K = 0
>> loop:
>>     pd = PartitionDispatch[K]
>>     idx = list/range_partition_for_tuple(pd->partdesc, ...)
>>     if (idx >= -1)
>>         return idx
>>     K = -(idx + 1)
>>
>> No recursion, minimal pointer chasing, no linked lists.  The whole
>> thing is basically trivial aside from the cost of
>> list/range_partition_for_tuple itself; optimizing that is a different
>> project.  I might have some details slightly off here, but hopefully
>> you can see what I'm going for: you want to keep the computation that
>> happens in get_partition_for_tuple() to an absolute minimum, and
>> instead set things up in advance so that getting the partition for a
>> tuple is FAST.  And you want the data structures that you are using in
>> that process to be very compact, hence arrays instead of linked lists.
>
> This sounds *much* better.  Here is a quick attempt at coding the design
> you have outlined above in the attached latest set of patches.

That shrank both 0006 and 0007 substantially, and it should be faster,
too.   I bet you can shrink them further:

- Why is PartitionKeyExecInfo a separate structure and why does it
have a NodeTag?  I bet you can dump the node tag, merge it into
PartitionDispatch, and save some more code and some more
pointer-chasing.

- I still think it's a seriously bad idea for list partitioning and
range partitioning to need different code-paths all over the place
here. List partitions support nulls but not multi-column partitioning
keys and range partitions support multi-column partitioning keys but
not nulls, but you could use an internal structure that supports both.
Then you wouldn't need partition_list_values_bsearch and also
partition_rbound_bsearch; you could have one kind of bound structure
that can be bsearch'd for either list or range.  You might even be
able to unify list_partition_for_tuple and range_partition_for_tuple
although that looks a little harder.  In either case, you bsearch for
the greatest value <= the value you have.  The only difference is that
for list partitioning, you have to enforce at the end that it is an
equal value, whereas for range partitioning less-than-or-equal-to is
enough.  But you should still be able to arrange for more code
sharing.

- I don't see why you need the bound->lower stuff any more.  If
rangeinfo.bounds[offset] is a lower bound for a partition, then
rangeinfo.bounds[offset+1] is either (a) the upper bound for that
partition and the partition is followed by a "gap" or (b) both the
upper bound for that partition and the lower bound for the next
partition.  With the inclusive/exclusive bound stuff gone, every range
bound has the same sense: if the probed value is <= the bound then
we're supposed to be a lower-numbered partition, but if > then we're
supposed to be in this partition or a higher-numbered one.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Updated patches attached.  I merged what used to be 0006 and 0007 into one.

On 2016/11/19 2:23, Robert Haas wrote:
> On Fri, Nov 18, 2016 at 5:59 AM, Amit Langote
> <Langote_Amit_f8@lab.ntt.co.jp> wrote:
>> Oh but wait, that means I can insert rows with NULLs in the range
>> partition key if I choose to insert it directly into the partition,
>> whereas I have been thinking all this while that there could never be
>> NULLs in the partition key of a range partition.  What's more,
>> get_qual_for_partbound() (patch 0003) emits a IS NOT NULL constraint for
>> every partition key column in case of a range partition.  Is that
>> wrongheaded altogether?  (also see my reply to your earlier message about
>> NULLs in the range partition key)
>
> The easiest thing to do might be to just enforce that all of the
> partition key columns have to be not-null when the range-partitioned
> table is defined, and reject any attempt to DROP NOT NULL on them
> later.  That's probably better that shoehorning it into the table
> constraint.

Agreed that forcing range partitioning columns to be NOT NULL during table
creation would be a better approach.  But then we would have to reject
using expressions in the range partition key, right?

>> Thanks for the idea below!
>>
>>> 1. Forget the idea of a tree.  Instead, let the total number of tables
>>> in the partitioning hierarchy be N and let the number of those that

[ ... ]

>>>
>>> No recursion, minimal pointer chasing, no linked lists.  The whole
>>> thing is basically trivial aside from the cost of
>>> list/range_partition_for_tuple itself; optimizing that is a different
>>> project.  I might have some details slightly off here, but hopefully
>>> you can see what I'm going for: you want to keep the computation that
>>> happens in get_partition_for_tuple() to an absolute minimum, and
>>> instead set things up in advance so that getting the partition for a
>>> tuple is FAST.  And you want the data structures that you are using in
>>> that process to be very compact, hence arrays instead of linked lists.
>>
>> This sounds *much* better.  Here is a quick attempt at coding the design
>> you have outlined above in the attached latest set of patches.
>
> That shrank both 0006 and 0007 substantially, and it should be faster,
> too.   I bet you can shrink them further:

Some changes described below have reduced the size to a certain degree.

>
> - Why is PartitionKeyExecInfo a separate structure and why does it
> have a NodeTag?  I bet you can dump the node tag, merge it into
> PartitionDispatch, and save some more code and some more
> pointer-chasing.

OK, I merged the fields of what used to be PartitionKeyExecInfo into
PartitionDispatchData as the latter's new fields key and keystate.

> - I still think it's a seriously bad idea for list partitioning and
> range partitioning to need different code-paths all over the place
> here. List partitions support nulls but not multi-column partitioning
> keys and range partitions support multi-column partitioning keys but
> not nulls, but you could use an internal structure that supports both.
> Then you wouldn't need partition_list_values_bsearch and also
> partition_rbound_bsearch; you could have one kind of bound structure
> that can be bsearch'd for either list or range.  You might even be
> able to unify list_partition_for_tuple and range_partition_for_tuple
> although that looks a little harder.  In either case, you bsearch for
> the greatest value <= the value you have.  The only difference is that
> for list partitioning, you have to enforce at the end that it is an
> equal value, whereas for range partitioning less-than-or-equal-to is
> enough.  But you should still be able to arrange for more code
> sharing.

I have considered these suggestions in the latest patch.  Now instead of
PartitionListInfo, PartitionRangeInfo, and BoundCollectionData structs,
there is only one PartitionBoundInfo which consolidates the partition
bound information of a partitioned table.  Some of the fields are
applicable only to one of list or range case; for example, null-accepting
list partition index, infinite status of individual range datums.

Also, there is now only one binary search function named
partition_bound_bsearch() which invokes a comparison function named
partition_bound_cmp().  The former searches a probe (a partition bound or
tuple) within a PartitionBoundInfo, which is passed all the way down to
the comparison function.

Also, we no longer have list_partition_for_tuple() and
range_partition_for_tuple().  Instead, in get_partition_for_tuple()
itself, there is a bsearch followed by list and range partitioning
specific steps based on the returned offset.

> - I don't see why you need the bound->lower stuff any more.  If
> rangeinfo.bounds[offset] is a lower bound for a partition, then
> rangeinfo.bounds[offset+1] is either (a) the upper bound for that
> partition and the partition is followed by a "gap" or (b) both the
> upper bound for that partition and the lower bound for the next
> partition.  With the inclusive/exclusive bound stuff gone, every range
> bound has the same sense: if the probed value is <= the bound then
> we're supposed to be a lower-numbered partition, but if > then we're
> supposed to be in this partition or a higher-numbered one.

OK, I've managed to get rid of lower.  At least it is no longer kept in
the new relcache struct PartitionBoundInfo.  It is still kept in
PartitionRangeBound which is used to hold individual range bounds when
sorting them (during relcache build).  Comparisons invoked during the
aforementioned sorting step still need to distinguish between lower and
upper bounds (such that '1)' < '[1').

Tuple-routing no longer needs to look at lower.  In that case, what you
described above applies.

As a result, one change became necessary: to how we flag individual range
bound datum as infinite or not.  Previously, it was a regular Boolean
value (either infinite or not) and to distinguish +infinity from
-infinity, we looked at whether the bound is lower or upper (the lower
flag).  Now, instead, the variable holding the status of individual range
bound datum is set to a ternary value: RANGE_DATUM_FINITE (0),
RANGE_DATUM_NEG_INF (1), and RANGE_DATUM_POS_INF (2), which still fits in
a bool.  Upon encountering an infinite range bound datum, whether it's
negative or positive infinity derives the comparison result.  Consider the
following example:

partition p1 from (1, unbounded) to (1, 1);
partition p2 from (1, 1) to (1, 10);
partition p3 from (1, 10) to (1, unbounded);
partition p4 from (2, unbounded) to (2, 1);
... so on

In this case, we need to be able to conclude, say, (1, -inf) < (1, 15) <
(1, +inf), so that tuple (1, 15) is assigned to the proper partition.

Does this last thing sound reasonable?

Thanks,
Amit

On Tue, Nov 22, 2016 at 4:15 AM, Amit Langote
<Langote_Amit_f8@lab.ntt.co.jp> wrote:
>> The easiest thing to do might be to just enforce that all of the
>> partition key columns have to be not-null when the range-partitioned
>> table is defined, and reject any attempt to DROP NOT NULL on them
>> later.  That's probably better that shoehorning it into the table
>> constraint.
>
> Agreed that forcing range partitioning columns to be NOT NULL during table
> creation would be a better approach.  But then we would have to reject
> using expressions in the range partition key, right?

Why?

> As a result, one change became necessary: to how we flag individual range
> bound datum as infinite or not.  Previously, it was a regular Boolean
> value (either infinite or not) and to distinguish +infinity from
> -infinity, we looked at whether the bound is lower or upper (the lower
> flag).  Now, instead, the variable holding the status of individual range
> bound datum is set to a ternary value: RANGE_DATUM_FINITE (0),
> RANGE_DATUM_NEG_INF (1), and RANGE_DATUM_POS_INF (2), which still fits in
> a bool.

You better not be using a bool to represent a ternary value!  Use an
enum for that -- or if in the system catalogs, a char.

> Upon encountering an infinite range bound datum, whether it's
> negative or positive infinity derives the comparison result.  Consider the
> following example:
>
> partition p1 from (1, unbounded) to (1, 1);
> partition p2 from (1, 1) to (1, 10);
> partition p3 from (1, 10) to (1, unbounded);
> partition p4 from (2, unbounded) to (2, 1);
> ... so on
>
> In this case, we need to be able to conclude, say, (1, -inf) < (1, 15) <
> (1, +inf), so that tuple (1, 15) is assigned to the proper partition.

Right.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

I am trying to create a partitioned table with primary keys on the
partitions. Here's the corresponding syntax as per documentation
CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [
IF NOT EXISTS ] table_name   PARTITION OF parent_table [ ( { column_name WITH OPTIONS [ column_constraint [ ... ] ]   |
table_constraint}   [, ... ]
 
) ] partition_bound_spec

IIUC, it should allow "create table t1_p1 partition of t1 (a primary
key) ...", (a primary key) is nothing but "column_name
column_constraint", but here's what happens
create table t1_p1 partition of t1 (a primary key) for values from (0) to (100);
ERROR:  syntax error at or near "primary"
LINE 1: create table t1_p1 partition of t1 (a primary key) for value...

The same syntax also suggests using table_constraints but that too doesn't workcreate table t1_p1 partition of t1
(primarykey (a) )  for values
 
from (0) to (100);
ERROR:  inherited relation "t1" is not a table or foreign table

of course t1 is a table, what it isn't?

Am I missing something? How do I define constraints on the partitions?


On Tue, Nov 22, 2016 at 2:45 PM, Amit Langote
<Langote_Amit_f8@lab.ntt.co.jp> wrote:
>
> Updated patches attached.  I merged what used to be 0006 and 0007 into one.
>
> On 2016/11/19 2:23, Robert Haas wrote:
>> On Fri, Nov 18, 2016 at 5:59 AM, Amit Langote
>> <Langote_Amit_f8@lab.ntt.co.jp> wrote:
>>> Oh but wait, that means I can insert rows with NULLs in the range
>>> partition key if I choose to insert it directly into the partition,
>>> whereas I have been thinking all this while that there could never be
>>> NULLs in the partition key of a range partition.  What's more,
>>> get_qual_for_partbound() (patch 0003) emits a IS NOT NULL constraint for
>>> every partition key column in case of a range partition.  Is that
>>> wrongheaded altogether?  (also see my reply to your earlier message about
>>> NULLs in the range partition key)
>>
>> The easiest thing to do might be to just enforce that all of the
>> partition key columns have to be not-null when the range-partitioned
>> table is defined, and reject any attempt to DROP NOT NULL on them
>> later.  That's probably better that shoehorning it into the table
>> constraint.
>
> Agreed that forcing range partitioning columns to be NOT NULL during table
> creation would be a better approach.  But then we would have to reject
> using expressions in the range partition key, right?
>
>>> Thanks for the idea below!
>>>
>>>> 1. Forget the idea of a tree.  Instead, let the total number of tables
>>>> in the partitioning hierarchy be N and let the number of those that
>
> [ ... ]
>
>>>>
>>>> No recursion, minimal pointer chasing, no linked lists.  The whole
>>>> thing is basically trivial aside from the cost of
>>>> list/range_partition_for_tuple itself; optimizing that is a different
>>>> project.  I might have some details slightly off here, but hopefully
>>>> you can see what I'm going for: you want to keep the computation that
>>>> happens in get_partition_for_tuple() to an absolute minimum, and
>>>> instead set things up in advance so that getting the partition for a
>>>> tuple is FAST.  And you want the data structures that you are using in
>>>> that process to be very compact, hence arrays instead of linked lists.
>>>
>>> This sounds *much* better.  Here is a quick attempt at coding the design
>>> you have outlined above in the attached latest set of patches.
>>
>> That shrank both 0006 and 0007 substantially, and it should be faster,
>> too.   I bet you can shrink them further:
>
> Some changes described below have reduced the size to a certain degree.
>
>>
>> - Why is PartitionKeyExecInfo a separate structure and why does it
>> have a NodeTag?  I bet you can dump the node tag, merge it into
>> PartitionDispatch, and save some more code and some more
>> pointer-chasing.
>
> OK, I merged the fields of what used to be PartitionKeyExecInfo into
> PartitionDispatchData as the latter's new fields key and keystate.
>
>> - I still think it's a seriously bad idea for list partitioning and
>> range partitioning to need different code-paths all over the place
>> here. List partitions support nulls but not multi-column partitioning
>> keys and range partitions support multi-column partitioning keys but
>> not nulls, but you could use an internal structure that supports both.
>> Then you wouldn't need partition_list_values_bsearch and also
>> partition_rbound_bsearch; you could have one kind of bound structure
>> that can be bsearch'd for either list or range.  You might even be
>> able to unify list_partition_for_tuple and range_partition_for_tuple
>> although that looks a little harder.  In either case, you bsearch for
>> the greatest value <= the value you have.  The only difference is that
>> for list partitioning, you have to enforce at the end that it is an
>> equal value, whereas for range partitioning less-than-or-equal-to is
>> enough.  But you should still be able to arrange for more code
>> sharing.
>
> I have considered these suggestions in the latest patch.  Now instead of
> PartitionListInfo, PartitionRangeInfo, and BoundCollectionData structs,
> there is only one PartitionBoundInfo which consolidates the partition
> bound information of a partitioned table.  Some of the fields are
> applicable only to one of list or range case; for example, null-accepting
> list partition index, infinite status of individual range datums.
>
> Also, there is now only one binary search function named
> partition_bound_bsearch() which invokes a comparison function named
> partition_bound_cmp().  The former searches a probe (a partition bound or
> tuple) within a PartitionBoundInfo, which is passed all the way down to
> the comparison function.
>
> Also, we no longer have list_partition_for_tuple() and
> range_partition_for_tuple().  Instead, in get_partition_for_tuple()
> itself, there is a bsearch followed by list and range partitioning
> specific steps based on the returned offset.
>
>> - I don't see why you need the bound->lower stuff any more.  If
>> rangeinfo.bounds[offset] is a lower bound for a partition, then
>> rangeinfo.bounds[offset+1] is either (a) the upper bound for that
>> partition and the partition is followed by a "gap" or (b) both the
>> upper bound for that partition and the lower bound for the next
>> partition.  With the inclusive/exclusive bound stuff gone, every range
>> bound has the same sense: if the probed value is <= the bound then
>> we're supposed to be a lower-numbered partition, but if > then we're
>> supposed to be in this partition or a higher-numbered one.
>
> OK, I've managed to get rid of lower.  At least it is no longer kept in
> the new relcache struct PartitionBoundInfo.  It is still kept in
> PartitionRangeBound which is used to hold individual range bounds when
> sorting them (during relcache build).  Comparisons invoked during the
> aforementioned sorting step still need to distinguish between lower and
> upper bounds (such that '1)' < '[1').
>
> Tuple-routing no longer needs to look at lower.  In that case, what you
> described above applies.
>
> As a result, one change became necessary: to how we flag individual range
> bound datum as infinite or not.  Previously, it was a regular Boolean
> value (either infinite or not) and to distinguish +infinity from
> -infinity, we looked at whether the bound is lower or upper (the lower
> flag).  Now, instead, the variable holding the status of individual range
> bound datum is set to a ternary value: RANGE_DATUM_FINITE (0),
> RANGE_DATUM_NEG_INF (1), and RANGE_DATUM_POS_INF (2), which still fits in
> a bool.  Upon encountering an infinite range bound datum, whether it's
> negative or positive infinity derives the comparison result.  Consider the
> following example:
>
> partition p1 from (1, unbounded) to (1, 1);
> partition p2 from (1, 1) to (1, 10);
> partition p3 from (1, 10) to (1, unbounded);
> partition p4 from (2, unbounded) to (2, 1);
> ... so on
>
> In this case, we need to be able to conclude, say, (1, -inf) < (1, 15) <
> (1, +inf), so that tuple (1, 15) is assigned to the proper partition.
>
> Does this last thing sound reasonable?
>
> Thanks,
> Amit



-- 
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company

Unsurprisingly ALTER TABLE worked
alter table t1_p1 add primary key(a);
ALTER TABLE
postgres=# \d+ t1_p1                                  Table "public.t1_p1"Column |  Type   | Collation | Nullable |
Default| Storage | Stats
 
target | Description
--------+---------+-----------+----------+---------+---------+--------------+-------------a      | integer |
|not null |         | plain   |              |b      | integer |           |          |         | plain   |
|
 
Indexes:   "t1_p1_pkey" PRIMARY KEY, btree (a)
Inherits: t1

On Thu, Nov 24, 2016 at 11:05 AM, Ashutosh Bapat
<ashutosh.bapat@enterprisedb.com> wrote:
> I am trying to create a partitioned table with primary keys on the
> partitions. Here's the corresponding syntax as per documentation
> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [
> IF NOT EXISTS ] table_name
>     PARTITION OF parent_table [ (
>   { column_name WITH OPTIONS [ column_constraint [ ... ] ]
>     | table_constraint }
>     [, ... ]
> ) ] partition_bound_spec
>
> IIUC, it should allow "create table t1_p1 partition of t1 (a primary
> key) ...", (a primary key) is nothing but "column_name
> column_constraint", but here's what happens
> create table t1_p1 partition of t1 (a primary key) for values from (0) to (100);
> ERROR:  syntax error at or near "primary"
> LINE 1: create table t1_p1 partition of t1 (a primary key) for value...
>
> The same syntax also suggests using table_constraints but that too doesn't work
>  create table t1_p1 partition of t1 (primary key (a) )  for values
> from (0) to (100);
> ERROR:  inherited relation "t1" is not a table or foreign table
>
> of course t1 is a table, what it isn't?
>
> Am I missing something? How do I define constraints on the partitions?
>
>
> On Tue, Nov 22, 2016 at 2:45 PM, Amit Langote
> <Langote_Amit_f8@lab.ntt.co.jp> wrote:
>>
>> Updated patches attached.  I merged what used to be 0006 and 0007 into one.
>>
>> On 2016/11/19 2:23, Robert Haas wrote:
>>> On Fri, Nov 18, 2016 at 5:59 AM, Amit Langote
>>> <Langote_Amit_f8@lab.ntt.co.jp> wrote:
>>>> Oh but wait, that means I can insert rows with NULLs in the range
>>>> partition key if I choose to insert it directly into the partition,
>>>> whereas I have been thinking all this while that there could never be
>>>> NULLs in the partition key of a range partition.  What's more,
>>>> get_qual_for_partbound() (patch 0003) emits a IS NOT NULL constraint for
>>>> every partition key column in case of a range partition.  Is that
>>>> wrongheaded altogether?  (also see my reply to your earlier message about
>>>> NULLs in the range partition key)
>>>
>>> The easiest thing to do might be to just enforce that all of the
>>> partition key columns have to be not-null when the range-partitioned
>>> table is defined, and reject any attempt to DROP NOT NULL on them
>>> later.  That's probably better that shoehorning it into the table
>>> constraint.
>>
>> Agreed that forcing range partitioning columns to be NOT NULL during table
>> creation would be a better approach.  But then we would have to reject
>> using expressions in the range partition key, right?
>>
>>>> Thanks for the idea below!
>>>>
>>>>> 1. Forget the idea of a tree.  Instead, let the total number of tables
>>>>> in the partitioning hierarchy be N and let the number of those that
>>
>> [ ... ]
>>
>>>>>
>>>>> No recursion, minimal pointer chasing, no linked lists.  The whole
>>>>> thing is basically trivial aside from the cost of
>>>>> list/range_partition_for_tuple itself; optimizing that is a different
>>>>> project.  I might have some details slightly off here, but hopefully
>>>>> you can see what I'm going for: you want to keep the computation that
>>>>> happens in get_partition_for_tuple() to an absolute minimum, and
>>>>> instead set things up in advance so that getting the partition for a
>>>>> tuple is FAST.  And you want the data structures that you are using in
>>>>> that process to be very compact, hence arrays instead of linked lists.
>>>>
>>>> This sounds *much* better.  Here is a quick attempt at coding the design
>>>> you have outlined above in the attached latest set of patches.
>>>
>>> That shrank both 0006 and 0007 substantially, and it should be faster,
>>> too.   I bet you can shrink them further:
>>
>> Some changes described below have reduced the size to a certain degree.
>>
>>>
>>> - Why is PartitionKeyExecInfo a separate structure and why does it
>>> have a NodeTag?  I bet you can dump the node tag, merge it into
>>> PartitionDispatch, and save some more code and some more
>>> pointer-chasing.
>>
>> OK, I merged the fields of what used to be PartitionKeyExecInfo into
>> PartitionDispatchData as the latter's new fields key and keystate.
>>
>>> - I still think it's a seriously bad idea for list partitioning and
>>> range partitioning to need different code-paths all over the place
>>> here. List partitions support nulls but not multi-column partitioning
>>> keys and range partitions support multi-column partitioning keys but
>>> not nulls, but you could use an internal structure that supports both.
>>> Then you wouldn't need partition_list_values_bsearch and also
>>> partition_rbound_bsearch; you could have one kind of bound structure
>>> that can be bsearch'd for either list or range.  You might even be
>>> able to unify list_partition_for_tuple and range_partition_for_tuple
>>> although that looks a little harder.  In either case, you bsearch for
>>> the greatest value <= the value you have.  The only difference is that
>>> for list partitioning, you have to enforce at the end that it is an
>>> equal value, whereas for range partitioning less-than-or-equal-to is
>>> enough.  But you should still be able to arrange for more code
>>> sharing.
>>
>> I have considered these suggestions in the latest patch.  Now instead of
>> PartitionListInfo, PartitionRangeInfo, and BoundCollectionData structs,
>> there is only one PartitionBoundInfo which consolidates the partition
>> bound information of a partitioned table.  Some of the fields are
>> applicable only to one of list or range case; for example, null-accepting
>> list partition index, infinite status of individual range datums.
>>
>> Also, there is now only one binary search function named
>> partition_bound_bsearch() which invokes a comparison function named
>> partition_bound_cmp().  The former searches a probe (a partition bound or
>> tuple) within a PartitionBoundInfo, which is passed all the way down to
>> the comparison function.
>>
>> Also, we no longer have list_partition_for_tuple() and
>> range_partition_for_tuple().  Instead, in get_partition_for_tuple()
>> itself, there is a bsearch followed by list and range partitioning
>> specific steps based on the returned offset.
>>
>>> - I don't see why you need the bound->lower stuff any more.  If
>>> rangeinfo.bounds[offset] is a lower bound for a partition, then
>>> rangeinfo.bounds[offset+1] is either (a) the upper bound for that
>>> partition and the partition is followed by a "gap" or (b) both the
>>> upper bound for that partition and the lower bound for the next
>>> partition.  With the inclusive/exclusive bound stuff gone, every range
>>> bound has the same sense: if the probed value is <= the bound then
>>> we're supposed to be a lower-numbered partition, but if > then we're
>>> supposed to be in this partition or a higher-numbered one.
>>
>> OK, I've managed to get rid of lower.  At least it is no longer kept in
>> the new relcache struct PartitionBoundInfo.  It is still kept in
>> PartitionRangeBound which is used to hold individual range bounds when
>> sorting them (during relcache build).  Comparisons invoked during the
>> aforementioned sorting step still need to distinguish between lower and
>> upper bounds (such that '1)' < '[1').
>>
>> Tuple-routing no longer needs to look at lower.  In that case, what you
>> described above applies.
>>
>> As a result, one change became necessary: to how we flag individual range
>> bound datum as infinite or not.  Previously, it was a regular Boolean
>> value (either infinite or not) and to distinguish +infinity from
>> -infinity, we looked at whether the bound is lower or upper (the lower
>> flag).  Now, instead, the variable holding the status of individual range
>> bound datum is set to a ternary value: RANGE_DATUM_FINITE (0),
>> RANGE_DATUM_NEG_INF (1), and RANGE_DATUM_POS_INF (2), which still fits in
>> a bool.  Upon encountering an infinite range bound datum, whether it's
>> negative or positive infinity derives the comparison result.  Consider the
>> following example:
>>
>> partition p1 from (1, unbounded) to (1, 1);
>> partition p2 from (1, 1) to (1, 10);
>> partition p3 from (1, 10) to (1, unbounded);
>> partition p4 from (2, unbounded) to (2, 1);
>> ... so on
>>
>> In this case, we need to be able to conclude, say, (1, -inf) < (1, 15) <
>> (1, +inf), so that tuple (1, 15) is assigned to the proper partition.
>>
>> Does this last thing sound reasonable?
>>
>> Thanks,
>> Amit
>
>
>
> --
> Best Wishes,
> Ashutosh Bapat
> EnterpriseDB Corporation
> The Postgres Database Company



-- 
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company

Hi Ashutosh,

On 2016/11/24 14:35, Ashutosh Bapat wrote:
> I am trying to create a partitioned table with primary keys on the
> partitions. Here's the corresponding syntax as per documentation
> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [
> IF NOT EXISTS ] table_name
>     PARTITION OF parent_table [ (
>   { column_name WITH OPTIONS [ column_constraint [ ... ] ]
>     | table_constraint }
>     [, ... ]
> ) ] partition_bound_spec
> 
> IIUC, it should allow "create table t1_p1 partition of t1 (a primary
> key) ...", (a primary key) is nothing but "column_name
> column_constraint", but here's what happens
> create table t1_p1 partition of t1 (a primary key) for values from (0) to (100);
> ERROR:  syntax error at or near "primary"
> LINE 1: create table t1_p1 partition of t1 (a primary key) for value...

You have to specify column constraints using the keywords WITH OPTIONS,
like below:

create table p1 partition of p (   a with options primary key
) for values in (1);

> The same syntax also suggests using table_constraints but that too doesn't work
>  create table t1_p1 partition of t1 (primary key (a) )  for values
> from (0) to (100);
> ERROR:  inherited relation "t1" is not a table or foreign table
> 
> of course t1 is a table, what it isn't?

It's a bug.  Forgot to consider RELKIND_PARTITIONED_TABLE to an if block
in the code that checks inheritance parent relation's relkind when
creating an index constraint (primary key) on a child table.  Will fix,
thanks for catching it.

Thanks,
Amit

On Thu, Nov 24, 2016 at 11:34 AM, Amit Langote
<Langote_Amit_f8@lab.ntt.co.jp> wrote:
>
> Hi Ashutosh,
>
> On 2016/11/24 14:35, Ashutosh Bapat wrote:
>> I am trying to create a partitioned table with primary keys on the
>> partitions. Here's the corresponding syntax as per documentation
>> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [
>> IF NOT EXISTS ] table_name
>>     PARTITION OF parent_table [ (
>>   { column_name WITH OPTIONS [ column_constraint [ ... ] ]
>>     | table_constraint }
>>     [, ... ]
>> ) ] partition_bound_spec
>>
>> IIUC, it should allow "create table t1_p1 partition of t1 (a primary
>> key) ...", (a primary key) is nothing but "column_name
>> column_constraint", but here's what happens
>> create table t1_p1 partition of t1 (a primary key) for values from (0) to (100);
>> ERROR:  syntax error at or near "primary"
>> LINE 1: create table t1_p1 partition of t1 (a primary key) for value...
>
> You have to specify column constraints using the keywords WITH OPTIONS,
> like below:
>
> create table p1 partition of p (
>     a with options primary key
> ) for values in (1);

Oh, sorry for not noticing it. You are right. Why do we need "with
option" there? Shouldn't user be able to specify just "a primary key";
it's not really an "option", it's a constraint.

>
>> The same syntax also suggests using table_constraints but that too doesn't work
>>  create table t1_p1 partition of t1 (primary key (a) )  for values
>> from (0) to (100);
>> ERROR:  inherited relation "t1" is not a table or foreign table
>>
>> of course t1 is a table, what it isn't?
>
> It's a bug.  Forgot to consider RELKIND_PARTITIONED_TABLE to an if block
> in the code that checks inheritance parent relation's relkind when
> creating an index constraint (primary key) on a child table.  Will fix,
> thanks for catching it.
>
> Thanks,
> Amit
>
>



-- 
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company

On 2016/11/24 15:10, Ashutosh Bapat wrote:
> On Thu, Nov 24, 2016 at 11:34 AM, Amit Langote wrote:
>> On 2016/11/24 14:35, Ashutosh Bapat wrote:
>>> IIUC, it should allow "create table t1_p1 partition of t1 (a primary
>>> key) ...", (a primary key) is nothing but "column_name
>>> column_constraint", but here's what happens
>>> create table t1_p1 partition of t1 (a primary key) for values from (0) to (100);
>>> ERROR:  syntax error at or near "primary"
>>> LINE 1: create table t1_p1 partition of t1 (a primary key) for value...
>>
>> You have to specify column constraints using the keywords WITH OPTIONS,
>> like below:
>>
>> create table p1 partition of p (
>>     a with options primary key
>> ) for values in (1);
> 
> Oh, sorry for not noticing it. You are right. Why do we need "with
> option" there? Shouldn't user be able to specify just "a primary key";
> it's not really an "option", it's a constraint.

I just adopted the existing syntax for specifying column/table constraints
of a table created with CREATE TABLE OF type_name.

Thanks,
Amit

On Thu, Nov 24, 2016 at 12:02 PM, Amit Langote
<Langote_Amit_f8@lab.ntt.co.jp> wrote:
> On 2016/11/24 15:10, Ashutosh Bapat wrote:
>> On Thu, Nov 24, 2016 at 11:34 AM, Amit Langote wrote:
>>> On 2016/11/24 14:35, Ashutosh Bapat wrote:
>>>> IIUC, it should allow "create table t1_p1 partition of t1 (a primary
>>>> key) ...", (a primary key) is nothing but "column_name
>>>> column_constraint", but here's what happens
>>>> create table t1_p1 partition of t1 (a primary key) for values from (0) to (100);
>>>> ERROR:  syntax error at or near "primary"
>>>> LINE 1: create table t1_p1 partition of t1 (a primary key) for value...
>>>
>>> You have to specify column constraints using the keywords WITH OPTIONS,
>>> like below:
>>>
>>> create table p1 partition of p (
>>>     a with options primary key
>>> ) for values in (1);
>>
>> Oh, sorry for not noticing it. You are right. Why do we need "with
>> option" there? Shouldn't user be able to specify just "a primary key";
>> it's not really an "option", it's a constraint.
>
> I just adopted the existing syntax for specifying column/table constraints
> of a table created with CREATE TABLE OF type_name.

Hmm, I don't fine it quite intuitive. But others might find it so.

-- 
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company

Hi Rushabh,

On 2016/11/22 22:11, Rushabh Lathia wrote:
> Hi Amit,
>
> I was just reading through your patches and here are some quick review
> comments
> for 0001-Catalog-and-DDL-for-partitioned-tables-17.patch.

Thanks for the review!

>
> Review comments for  0001-Catalog-and-DDL-for-partitioned-tables-17.patch:
>
> 1)
> @@ -1102,9 +1104,10 @@ heap_create_with_catalog(const char *relname,
>      {
>          /* Use binary-upgrade override for pg_class.oid/relfilenode? */
>          if (IsBinaryUpgrade &&
> -            (relkind == RELKIND_RELATION || relkind == RELKIND_SEQUENCE ||
> -             relkind == RELKIND_VIEW || relkind == RELKIND_MATVIEW ||
> -             relkind == RELKIND_COMPOSITE_TYPE || relkind ==
> RELKIND_FOREIGN_TABLE))
> +            (relkind == RELKIND_RELATION || relkind ==
> RELKIND_PARTITIONED_TABLE ||
> +             relkind == RELKIND_SEQUENCE || relkind == RELKIND_VIEW ||
> +             relkind == RELKIND_MATVIEW || relkind ==
> RELKIND_COMPOSITE_TYPE ||
> +             relkind == RELKIND_FOREIGN_TABLE))
>
> You should add the RELKIND_PARTITIONED_TABLE at the end of if condition that
> will make diff minimal. While reading through the patch I noticed that there
> is inconsistency - someplace its been added at the end and at few places its
> at the start. I think you can make add it at the end of condition and be
> consistent with each place.

OK, done.

>
> 2)
>
> +        /*
> +         * We need to transform the raw parsetrees corresponding to
> partition
> +         * expressions into executable expression trees.  Like column
> defaults
> +         * and CHECK constraints, we could not have done the transformation
> +         * earlier.
> +         */
>
>
> Additional space before "Like column defaults".

I think it's a common practice to add two spaces after a sentence-ending
period [https://www.gnu.org/prep/standards/html_node/Comments.html], which
it seems, is followed more or less regularly in the formatting of the
comments in the PostgreSQL source code.

> 3)
> -    char        relkind;
> +    char        relkind,
> +                expected_relkind;
>
> Newly added variable should be define separately with its type. Something
> like:
>
>     char        relkind;
> +    char        expected_relkind;

OK, done.

>
> 4)
>
> a)
> +    /* Prevent partitioned tables from becoming inheritance parents */
> +    if (parent_rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
> +        ereport(ERROR,
> +                (errcode(ERRCODE_WRONG_OBJECT_TYPE),
> +                 errmsg("cannot inherit from partitioned table \"%s\"",
> +                         parent->relname)));
> +
>
> need alignment for last line.

Fixed.

> b)
> +            atttuple = SearchSysCacheAttName(RelationGetRelid(rel),
> pelem->name);
> +            if (!HeapTupleIsValid(atttuple))
> +                ereport(ERROR,
> +                        (errcode(ERRCODE_UNDEFINED_COLUMN),
> +                         errmsg("column \"%s\" named in partition key does
> not exist",
> +                         pelem->name)));
> +            attform = (Form_pg_attribute) GETSTRUCT(atttuple);
> +
> +            if (attform->attnum <= 0)
> +                ereport(ERROR,
> +                        (errcode(ERRCODE_UNDEFINED_COLUMN),
> +                         errmsg("cannot use system column \"%s\" in
> partition key",
> +                         pelem->name)));
>
> need alignment for last line of ereport

Fixed.

> c)
> +        /* Disallow ROW triggers on partitioned tables */
> +        if (stmt->row && rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
> +            ereport(ERROR,
> +                    (errcode(ERRCODE_WRONG_OBJECT_TYPE),
> +                    errmsg("\"%s\" is a partitioned table",
> +                            RelationGetRelationName(rel)),
> +              errdetail("Partitioned tables cannot have ROW triggers.")));
>
> need alignment

Fixed.

> 5)
>
> @@ -2512,6 +2579,7 @@ transformAlterTableStmt(Oid relid, AlterTableStmt
> *stmt,
>      cxt.blist = NIL;
>      cxt.alist = NIL;
>      cxt.pkey = NULL;
> +    cxt.ispartitioned = rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE;
>
> I think adding bracket will look code more clear.
>
> +    cxt.ispartitioned = (rel->rd_rel->relkind ==
> RELKIND_PARTITIONED_TABLE);

Agreed, done.

> 6)
>
> + * RelationBuildPartitionKey
> + *        Build and attach to relcache partition key data of relation
> + *
> + * Partitioning key data is stored in CacheMemoryContext to ensure it
> survives
> + * as long as the relcache.  To avoid leaking memory in that context in
> case
> + * of an error partway through this function, we build the structure in the
> + * working context (which must be short-lived) and copy the completed
> + * structure into the cache memory.
>
> extra space before "To avoid leaking memory"

Same thing I said above.

> 7)
> +    /* variable-length fields start here, but we allow direct access to
> partattrs */
> +    int2vector        partattrs;        /* attribute numbers of columns in
> the
>
> Why partattrs is allow direct access - its not really clear from the
> comments.

I have copied the comment from another catalog header file (a number of
catalog headers have the same comment).  I updated the new file's comment
to say a little bit more; I wonder if the comment should be updated in
other files as well?  However, I noticed that there are explanatory notes
elsewhere (for example, around the code that reads such a field from the
catalog) about why the first variable-length field of a catalog's tuple
(especially of type int2vector or oidvector) are directly accessible via
their C struct offsets.

> I will continue reading more patch and testing functionality.. will share
> the
> comments as I have it.

Updated patches attached.  I have also considered Robert's comments [1] as
follows and fixed a bug that Ashutosh reported [2]:

- Force partition key columns to be NOT NULL at the table creation time
  if using range partitioning
- Use enum to represent the content of individual range datums (viz.
  finite datum, -infinity, +infinity) in the relcache struct

Thanks,
Amit

[1]
https://www.postgresql.org/message-id/CA+TgmoZ-feQsxc7U_JerM_AFChp3Qf6btK708SAe7M8Vdv5=jA@mail.gmail.com
[2]
https://www.postgresql.org/message-id/306c85e9-c702-3742-eeff-9b7a40498afc%40lab.ntt.co.jp

On 2016/11/23 4:50, Robert Haas wrote:
> On Tue, Nov 22, 2016 at 4:15 AM, Amit Langote
> <Langote_Amit_f8@lab.ntt.co.jp> wrote:
>>> The easiest thing to do might be to just enforce that all of the
>>> partition key columns have to be not-null when the range-partitioned
>>> table is defined, and reject any attempt to DROP NOT NULL on them
>>> later.  That's probably better that shoehorning it into the table
>>> constraint.
>>
>> Agreed that forcing range partitioning columns to be NOT NULL during table
>> creation would be a better approach.  But then we would have to reject
>> using expressions in the range partition key, right?
> 
> Why?

I was thinking of it like how primary key columns cannot contain
expressions; the reason for which, I assume, is because while we can
ensure that a column contains only non-null values by defining a
constraint on the column, there is no way to force expressions to be non-null.

In any case, I have implemented in the latest patch that when creating a
range partitioned table, its partition key columns are automatically set
to be NOT NULL.  Although, it leaves out the columns that are referenced
in expressions.  So even after doing so, we need to check after computing
the range partition key of a input row, that none of the partitions keys
is null, because expressions can still return null.

Also, it does nothing to help the undesirable situation that one can
insert a row with a null partition key (expression) into any of the range
partitions if targeted directly, because of how ExecQual() handles
nullable constraint expressions (treats null value as satisfying the
partition constraint).

An alternative possibly worth considering might be to somehow handle the
null range partition keys within the logic to compare against range bound
datums.  It looks like other databases will map the rows containing nulls
to the unbounded partition.  One database allows specifying NULLS
FIRST/LAST and maps a row containing null key to the partition with
-infinity as the lower bound or +infinity as the upper bound, respectively
with  NULLS LAST the default behavior.

In our case, if we allowed a similar way of defining a range partitioned
table:

create table p (a int, b int) partition by range nulls first (a);
create table p0 partition of p for values from (unbounded) to (1);
create table p1 partition of p for values from (1) to (10);
create table p2 partition of p for values from (10) to (unbounded);

Row (null, 1) will be mapped to p0.  If we didn't have p0, we would report
the "partition not found" error.

In case of a multi-column key:

create table p (a int, b int) partition by range (a, b);
create table p0 partition of p for values from (1, unbounded) to (1, 1);
create table p1 partition of p for values from (1, 1) to (1, 10);
create table p2 partition of p for values from (1, 10) to (1, unbounded);

Row (1, null) will be mapped to p2 (default nulls last behavior).

But I guess we still end up without a solution for the problem that a row
with null partition key (expression) could be inserted into any of the
range partitions if targeted directly.

Thoughts?

>> As a result, one change became necessary: to how we flag individual range
>> bound datum as infinite or not.  Previously, it was a regular Boolean
>> value (either infinite or not) and to distinguish +infinity from
>> -infinity, we looked at whether the bound is lower or upper (the lower
>> flag).  Now, instead, the variable holding the status of individual range
>> bound datum is set to a ternary value: RANGE_DATUM_FINITE (0),
>> RANGE_DATUM_NEG_INF (1), and RANGE_DATUM_POS_INF (2), which still fits in
>> a bool.
> 
> You better not be using a bool to represent a ternary value!  Use an
> enum for that -- or if in the system catalogs, a char.

OK, created an enum called RangeDatumContent.  In the system catalog, we
still store the boolean value; it is only after we read it into the
relcache structure that we use one of these enum values.  I'm worried
though that using enum would consume more memory (we need to store nparts
* partnattrs instances of the enum).

Thanks,
Amit

Amit Langote wrote:
> On 2016/11/24 15:10, Ashutosh Bapat wrote:
> > On Thu, Nov 24, 2016 at 11:34 AM, Amit Langote wrote:

> >> You have to specify column constraints using the keywords WITH OPTIONS,
> >> like below:
> >>
> >> create table p1 partition of p (
> >>     a with options primary key
> >> ) for values in (1);
> > 
> > Oh, sorry for not noticing it. You are right. Why do we need "with
> > option" there? Shouldn't user be able to specify just "a primary key";
> > it's not really an "option", it's a constraint.
> 
> I just adopted the existing syntax for specifying column/table constraints
> of a table created with CREATE TABLE OF type_name.

I think CREATE TABLE OF is pretty much a corner case.  I agree that
allowing the constraint right after the constraint name is more
intuitive.

-- 
Álvaro Herrera                https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On 2016/11/25 4:36, Alvaro Herrera wrote:
> Amit Langote wrote:
>> On 2016/11/24 15:10, Ashutosh Bapat wrote:
>>> On Thu, Nov 24, 2016 at 11:34 AM, Amit Langote wrote:
> 
>>>> You have to specify column constraints using the keywords WITH OPTIONS,
>>>> like below:
>>>>
>>>> create table p1 partition of p (
>>>>     a with options primary key
>>>> ) for values in (1);
>>>
>>> Oh, sorry for not noticing it. You are right. Why do we need "with
>>> option" there? Shouldn't user be able to specify just "a primary key";
>>> it's not really an "option", it's a constraint.
>>
>> I just adopted the existing syntax for specifying column/table constraints
>> of a table created with CREATE TABLE OF type_name.
> 
> I think CREATE TABLE OF is pretty much a corner case.  I agree that
> allowing the constraint right after the constraint name is more
> intuitive.

I assume you meant "...right after the column name"?

I will modify the grammar to allow that way then, so that the following
will work:

create table p1 partition of p (    a primary key
) for values in (1);

Thanks,
Amit

On Thu, Nov 24, 2016 at 6:13 AM, Amit Langote
<Langote_Amit_f8@lab.ntt.co.jp> wrote:
> On 2016/11/23 4:50, Robert Haas wrote:
>> On Tue, Nov 22, 2016 at 4:15 AM, Amit Langote
>> <Langote_Amit_f8@lab.ntt.co.jp> wrote:
>>>> The easiest thing to do might be to just enforce that all of the
>>>> partition key columns have to be not-null when the range-partitioned
>>>> table is defined, and reject any attempt to DROP NOT NULL on them
>>>> later.  That's probably better that shoehorning it into the table
>>>> constraint.
>>>
>>> Agreed that forcing range partitioning columns to be NOT NULL during table
>>> creation would be a better approach.  But then we would have to reject
>>> using expressions in the range partition key, right?
>>
>> Why?
>
> I was thinking of it like how primary key columns cannot contain
> expressions; the reason for which, I assume, is because while we can
> ensure that a column contains only non-null values by defining a
> constraint on the column, there is no way to force expressions to be non-null.
>
> In any case, I have implemented in the latest patch that when creating a
> range partitioned table, its partition key columns are automatically set
> to be NOT NULL.  Although, it leaves out the columns that are referenced
> in expressions.  So even after doing so, we need to check after computing
> the range partition key of a input row, that none of the partitions keys
> is null, because expressions can still return null.

Right.  And ensuring that those columns were NOT NULL would be wrong,
as it wouldn't guarantee a non-null result anyway.

> Also, it does nothing to help the undesirable situation that one can
> insert a row with a null partition key (expression) into any of the range
> partitions if targeted directly, because of how ExecQual() handles
> nullable constraint expressions (treats null value as satisfying the
> partition constraint).

That's going to have to be fixed somehow.  How bad would it be if we
passed ExecQual's third argument as false for partition constraints?
Or else you could generate the actual constraint as expr IS NOT NULL
AND expr >= lb AND expr < ub.

> An alternative possibly worth considering might be to somehow handle the
> null range partition keys within the logic to compare against range bound
> datums.  It looks like other databases will map the rows containing nulls
> to the unbounded partition.  One database allows specifying NULLS
> FIRST/LAST and maps a row containing null key to the partition with
> -infinity as the lower bound or +infinity as the upper bound, respectively
> with  NULLS LAST the default behavior.

It seems more future-proof not to allow NULLs at all for now, and
figure out what if anything we want to do about that later.  I mean,
with the syntax we've got here, anything else is basically deciding
whether NULL is the lowest value or the highest value.  It would be
convenient for my employer if we made the same decision that Oracle
did, here, but it doesn't really seem like the PostgreSQL way - or to
put that another way, it's really ugly and unprincipled.  So I
recommend we decide for now that a partitioning column can't be null
and a partitioning expression can't evaluate to NULL.  If it does,
ERROR.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Amit Langote wrote:
> On 2016/11/25 4:36, Alvaro Herrera wrote:

> > I think CREATE TABLE OF is pretty much a corner case.  I agree that
> > allowing the constraint right after the constraint name is more
> > intuitive.
> 
> I assume you meant "...right after the column name"?

Eh, right.


-- 
Álvaro Herrera                https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

>
> I assume you meant "...right after the column name"?
>
> I will modify the grammar to allow that way then, so that the following
> will work:
>
> create table p1 partition of p (
>      a primary key
> ) for values in (1);
>

That seems to be non-intuitive as well. The way it's written it looks
like "a primary key" is associated with p rather than p1.

Is there any column constraint that can not be a table constraint? If
no, then we can think of dropping column constraint syntax all
together and let the user specify column constraints through table
constraint syntax. OR we may drop constraints all-together from the
"CREATE TABLE .. PARTITION OF" syntax and let user handle it through
ALTER TABLE commands. In a later version, we will introduce constraint
syntax in that DDL.

-- 
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company

On 2016/11/25 13:51, Ashutosh Bapat wrote:
>>
>> I assume you meant "...right after the column name"?
>>
>> I will modify the grammar to allow that way then, so that the following
>> will work:
>>
>> create table p1 partition of p (
>>      a primary key
>> ) for values in (1);
>>
> 
> That seems to be non-intuitive as well. The way it's written it looks
> like "a primary key" is associated with p rather than p1.

It kind of does, but it's still a "create table p1" statement, so it's not
that ambiguous, IMHO.  Although I'm not attached to this syntax, there may
not be other options, such as moving the parenthesized list right before
"partition of", due to resulting grammar conflicts.

> Is there any column constraint that can not be a table constraint?

NOT NULL, DEFAULT, and COLLATE (although only the first of these is really
a constraint, albeit without a pg_constraint entry)

> If no, then we can think of dropping column constraint syntax all
> together and let the user specify column constraints through table
> constraint syntax. OR we may drop constraints all-together from the
> "CREATE TABLE .. PARTITION OF" syntax and let user handle it through
> ALTER TABLE commands. In a later version, we will introduce constraint
> syntax in that DDL.

Hmm, it was like that in the past versions of the patch, but I thought
it'd be better to follow the CREATE TABLE OF style to allow specifying the
table and column constraints during table creation time.  If many think
that it is not required (or should have some other syntax), I will modify
the patch accordingly.

Thanks,
Amit

On 2016/11/25 11:44, Robert Haas wrote:
> On Thu, Nov 24, 2016 at 6:13 AM, Amit Langote wrote:
>> Also, it does nothing to help the undesirable situation that one can
>> insert a row with a null partition key (expression) into any of the range
>> partitions if targeted directly, because of how ExecQual() handles
>> nullable constraint expressions (treats null value as satisfying the
>> partition constraint).
> 
> That's going to have to be fixed somehow.  How bad would it be if we
> passed ExecQual's third argument as false for partition constraints?
> Or else you could generate the actual constraint as expr IS NOT NULL
> AND expr >= lb AND expr < ub.

About the former, I think that might work.  If a column is NULL, it would
be caught in ExecConstraints() even before ExecQual() is called, because
of the NOT NULL constraint.  If an expression is NULL, or for some reason,
the partitioning operator (=, >=, or <) returned NULL even for a non-NULL
column or expression, then ExecQual() would fail if we passed false for
resultForNull.  Not sure if that would be violating the SQL specification
though.

The latter would work too.  But I guess we would only emit expr IS NOT
NULL, not column IS NOT NULL, because columns are covered by NOT NULL
constraints.

>> An alternative possibly worth considering might be to somehow handle the
>> null range partition keys within the logic to compare against range bound
>> datums.  It looks like other databases will map the rows containing nulls
>> to the unbounded partition.  One database allows specifying NULLS
>> FIRST/LAST and maps a row containing null key to the partition with
>> -infinity as the lower bound or +infinity as the upper bound, respectively
>> with  NULLS LAST the default behavior.
> 
> It seems more future-proof not to allow NULLs at all for now, and
> figure out what if anything we want to do about that later.  I mean,
> with the syntax we've got here, anything else is basically deciding
> whether NULL is the lowest value or the highest value.  It would be
> convenient for my employer if we made the same decision that Oracle
> did, here, but it doesn't really seem like the PostgreSQL way - or to
> put that another way, it's really ugly and unprincipled.  So I
> recommend we decide for now that a partitioning column can't be null
> and a partitioning expression can't evaluate to NULL.  If it does,
> ERROR.

OK, we can decide later if we want to handle NULLs somehow.

Thanks,
Amit

Here are some comments on 0003 patch.
1. In ALTER TABLE documentation we should refer to CREATE TABLE documentation
for partition_bound_spec syntax, similar ADD table_constraint note.

2. I think, "of the target table" is missing after all the ... constraints
+      match.  Also, it must have all the <literal>NOT NULL</literal> and
+      <literal>CHECK</literal> constraints present in the target table.

3. I think, using "any of the" instead of "some" is preferred in the following
sentence. In the following sentence, we should use "such a constraint" instead
of "constraints" to avoid mixed singular and plural usage.
+      If some <literal>CHECK</literal> constraint of the table being attached

4. This construction is ambiguous. "are not considered" - considered where? for
what? Constraints on which object?
+      clause.  Currently <literal>UNIQUE</literal>, <literal>PRIMARY
KEY</literal>,
+      and <literal>FOREIGN KEY</literal> constraints are not considered.

5. What is a partition constraint? Do we define that term anywhere in the
documentation? If not we should define that term and then use it here.
+      A full table scan is performed on the table being attached to check that
+      no existing row in the table violates the partition constraint.  It is

6. The paragraph following
+      A full table scan is performed on the table
being attached to check that seems to be confusing. It says that the
potentially expensive scan can be avoided by adding a constraint equivalent to
partition constraint. That seems to be wrong. The table will be scanned anyway,
when adding a valid constraint per ALTER TABLE ADD table_constraint
documentation. Instead you might want to say that the table scan will not be
carried out if the existing constraints imply the partition constraints.

7. You might want to specify the fate of range or lists covered by the
partition being detached in DETACH PARTITION section.

8. Since partition bound specification is more than a few words, you might want
to refer to the actual syntax in CREATE TABLE command.
+      <term><replaceable
class="PARAMETER">partition_bound_spec</replaceable></term>
+      <listitem>
+       <para>
+        The partition bound specification for a new partition.
+       </para>
+      </listitem>
+     </varlistentry>

9. I think, we need to use "may be scanned" instead of "is scanned", given that
some other part of the documentation talks about "avoiding" the table scan. May
be refer to that documentation to clarify the uncertainty.
+    Similarly, when attaching a new partition it is scanned to verify that

10. The same paragraph talks about multiple ALTER TABLE commands being run
together to avoid multiple table rewrites. Do we want to document whether
ATTACH/DETACH partition can be run with any other command or not.

11. ATTACH partition documentation is not talking about the unlogged or
temporary tables being attached to is logged or non-temporary. What is current
code doing about these cases? Do we allow such mixed partition hierarchy?
+    existing rows meet the partition constraint.

I will continue to look at the patch.

On Thu, Nov 24, 2016 at 4:04 PM, Amit Langote
<Langote_Amit_f8@lab.ntt.co.jp> wrote:
>
> Hi Rushabh,
>
> On 2016/11/22 22:11, Rushabh Lathia wrote:
>> Hi Amit,
>>
>> I was just reading through your patches and here are some quick review
>> comments
>> for 0001-Catalog-and-DDL-for-partitioned-tables-17.patch.
>
> Thanks for the review!
>
>>
>> Review comments for  0001-Catalog-and-DDL-for-partitioned-tables-17.patch:
>>
>> 1)
>> @@ -1102,9 +1104,10 @@ heap_create_with_catalog(const char *relname,
>>      {
>>          /* Use binary-upgrade override for pg_class.oid/relfilenode? */
>>          if (IsBinaryUpgrade &&
>> -            (relkind == RELKIND_RELATION || relkind == RELKIND_SEQUENCE ||
>> -             relkind == RELKIND_VIEW || relkind == RELKIND_MATVIEW ||
>> -             relkind == RELKIND_COMPOSITE_TYPE || relkind ==
>> RELKIND_FOREIGN_TABLE))
>> +            (relkind == RELKIND_RELATION || relkind ==
>> RELKIND_PARTITIONED_TABLE ||
>> +             relkind == RELKIND_SEQUENCE || relkind == RELKIND_VIEW ||
>> +             relkind == RELKIND_MATVIEW || relkind ==
>> RELKIND_COMPOSITE_TYPE ||
>> +             relkind == RELKIND_FOREIGN_TABLE))
>>
>> You should add the RELKIND_PARTITIONED_TABLE at the end of if condition that
>> will make diff minimal. While reading through the patch I noticed that there
>> is inconsistency - someplace its been added at the end and at few places its
>> at the start. I think you can make add it at the end of condition and be
>> consistent with each place.
>
> OK, done.
>
>>
>> 2)
>>
>> +        /*
>> +         * We need to transform the raw parsetrees corresponding to
>> partition
>> +         * expressions into executable expression trees.  Like column
>> defaults
>> +         * and CHECK constraints, we could not have done the transformation
>> +         * earlier.
>> +         */
>>
>>
>> Additional space before "Like column defaults".
>
> I think it's a common practice to add two spaces after a sentence-ending
> period [https://www.gnu.org/prep/standards/html_node/Comments.html], which
> it seems, is followed more or less regularly in the formatting of the
> comments in the PostgreSQL source code.
>
>> 3)
>> -    char        relkind;
>> +    char        relkind,
>> +                expected_relkind;
>>
>> Newly added variable should be define separately with its type. Something
>> like:
>>
>>     char        relkind;
>> +    char        expected_relkind;
>
> OK, done.
>
>>
>> 4)
>>
>> a)
>> +    /* Prevent partitioned tables from becoming inheritance parents */
>> +    if (parent_rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
>> +        ereport(ERROR,
>> +                (errcode(ERRCODE_WRONG_OBJECT_TYPE),
>> +                 errmsg("cannot inherit from partitioned table \"%s\"",
>> +                         parent->relname)));
>> +
>>
>> need alignment for last line.
>
> Fixed.
>
>> b)
>> +            atttuple = SearchSysCacheAttName(RelationGetRelid(rel),
>> pelem->name);
>> +            if (!HeapTupleIsValid(atttuple))
>> +                ereport(ERROR,
>> +                        (errcode(ERRCODE_UNDEFINED_COLUMN),
>> +                         errmsg("column \"%s\" named in partition key does
>> not exist",
>> +                         pelem->name)));
>> +            attform = (Form_pg_attribute) GETSTRUCT(atttuple);
>> +
>> +            if (attform->attnum <= 0)
>> +                ereport(ERROR,
>> +                        (errcode(ERRCODE_UNDEFINED_COLUMN),
>> +                         errmsg("cannot use system column \"%s\" in
>> partition key",
>> +                         pelem->name)));
>>
>> need alignment for last line of ereport
>
> Fixed.
>
>> c)
>> +        /* Disallow ROW triggers on partitioned tables */
>> +        if (stmt->row && rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
>> +            ereport(ERROR,
>> +                    (errcode(ERRCODE_WRONG_OBJECT_TYPE),
>> +                    errmsg("\"%s\" is a partitioned table",
>> +                            RelationGetRelationName(rel)),
>> +              errdetail("Partitioned tables cannot have ROW triggers.")));
>>
>> need alignment
>
> Fixed.
>
>> 5)
>>
>> @@ -2512,6 +2579,7 @@ transformAlterTableStmt(Oid relid, AlterTableStmt
>> *stmt,
>>      cxt.blist = NIL;
>>      cxt.alist = NIL;
>>      cxt.pkey = NULL;
>> +    cxt.ispartitioned = rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE;
>>
>> I think adding bracket will look code more clear.
>>
>> +    cxt.ispartitioned = (rel->rd_rel->relkind ==
>> RELKIND_PARTITIONED_TABLE);
>
> Agreed, done.
>
>> 6)
>>
>> + * RelationBuildPartitionKey
>> + *        Build and attach to relcache partition key data of relation
>> + *
>> + * Partitioning key data is stored in CacheMemoryContext to ensure it
>> survives
>> + * as long as the relcache.  To avoid leaking memory in that context in
>> case
>> + * of an error partway through this function, we build the structure in the
>> + * working context (which must be short-lived) and copy the completed
>> + * structure into the cache memory.
>>
>> extra space before "To avoid leaking memory"
>
> Same thing I said above.
>
>> 7)
>> +    /* variable-length fields start here, but we allow direct access to
>> partattrs */
>> +    int2vector        partattrs;        /* attribute numbers of columns in
>> the
>>
>> Why partattrs is allow direct access - its not really clear from the
>> comments.
>
> I have copied the comment from another catalog header file (a number of
> catalog headers have the same comment).  I updated the new file's comment
> to say a little bit more; I wonder if the comment should be updated in
> other files as well?  However, I noticed that there are explanatory notes
> elsewhere (for example, around the code that reads such a field from the
> catalog) about why the first variable-length field of a catalog's tuple
> (especially of type int2vector or oidvector) are directly accessible via
> their C struct offsets.
>
>> I will continue reading more patch and testing functionality.. will share
>> the
>> comments as I have it.
>
> Updated patches attached.  I have also considered Robert's comments [1] as
> follows and fixed a bug that Ashutosh reported [2]:
>
> - Force partition key columns to be NOT NULL at the table creation time
>   if using range partitioning
> - Use enum to represent the content of individual range datums (viz.
>   finite datum, -infinity, +infinity) in the relcache struct
>
> Thanks,
> Amit
>
> [1]
> https://www.postgresql.org/message-id/CA+TgmoZ-feQsxc7U_JerM_AFChp3Qf6btK708SAe7M8Vdv5=jA@mail.gmail.com
> [2]
> https://www.postgresql.org/message-id/306c85e9-c702-3742-eeff-9b7a40498afc%40lab.ntt.co.jp



-- 
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company

On 2016/11/17 20:27, Amit Langote wrote:
> On 2016/11/16 4:21, Robert Haas wrote:
>> Have you done any performance testing on the tuple routing code?
>> Suppose we insert a million (or 10 million) tuples into an
>> unpartitioned table, a table with 10 partitions, a table with 100
>> partitions, a table with 1000 partitions, and a table that is
>> partitioned into 10 partitions each of which has 10 subpartitions.
>> Ideally, the partitioned cases would run almost as fast as the
>> unpartitioned case, but probably there will be some overhead.
>> However, it would be useful to know how much.  Also, it would be
>> useful to set up the same cases with inheritance using a PL/pgsql ON
>> INSERT trigger for tuple routing and compare.  Hopefully the tuple
>> routing code is far faster than a trigger, but we should make sure
>> that's the case and look for optimizations if not.  Also, it would be
>> useful to know how much slower the tuple-mapping-required case is than
>> the no-tuple-mapping-required case.
> 
> OK, I will share the performance results soon.

Sorry about the delay; here are some numbers with the following
partitioning schema:

# plain table
create table plain (a date, b int, c int);

# partitioned table
create table ptab (a date, b int, c int) partition by range (a, b);

Partitions (not the full commands):

ptab_00001 for values from ('2016-11-29',    1) to ('2016-11-29', 1000);
ptab_00002 for values from ('2016-11-29', 1000) to ('2016-11-29', 2000);
...
ptab_00005 for values from ('2016-11-29', 4000) to ('2016-11-29', 5000);

ptab_00006 for values from ('2016-11-30',    1) to ('2016-11-30', 1000);
...
...
ptab_NNNNN for values from ('20XX-XX-XX', 4000) to ('20XX-XX-XX', 5000);

# inheritance partitioned table
create table itab (a date, b int, c int);
create table itab_00001 (   check part_check check (a = '2016-11-29' and b >=    1 and b < 1000)
) inherits (itab);
...
create table itab_00005 (   check part_check check (a = '2016-11-29' and b >= 4000 and b < 5000)
) inherits (itab);
create table itab_0006 (   check part_check check (a = '2016-11-30' and b >=    1and b < 1000)
) inherits (itab);
...
...
create table itab_NNNNN (   check part_check check (a = '2016-11-29' and b >= 4000 and b < 5000)
) inherits (itab);


The BR trigger (on itab) procedure as follows:

CREATE OR REPLACE FUNCTION itab_ins_trig()
RETURNS TRIGGER AS $$
DECLARE partno text;
BEGIN SELECT to_char((NEW.a - '2016-11-29'::date) * 5 + NEW.b / 1000 + 1,                'fm00000') INTO partno;
EXECUTE'INSERT INTO itab_' || partno || ' SELECT $1.*' USING NEW; RETURN NULL;
 
END; $$ LANGUAGE plpgsql;

Note that the tuple-routing procedure above assumes a fixed-stride range
partitioning scheme (shown as tg-direct-map below).  In other cases, the
simplest approach involves defining a if-else ladder, which I tried too
(shown as tg-if-else below), but reporting times only for up to 200
partitions at most (I'm sure there might be ways to be smarter there
somehow, but I didn't; the point here may only be to compare the new
tuple-routing code's overhead vs. trigger overhead in the traditional method).

# All times in seconds (on my modestly-powerful development VM)
#
# nrows = 10,000,000 generated using:
#
# INSERT INTO $tab
# SELECT '$last'::date - ((s.id % $maxsecs + 1)::bigint || 's')::interval,
#       (random() * 5000)::int % 4999 + 1,
#        case s.id % 10
#          when 0 then 'a'
#          when 1 then 'b'
#          when 2 then 'c'
#          ...
#          when 9 then 'j'
#       end
# FROM generate_series(1, $nrows) s(id)
# ORDER BY random();
#
# The first item in the select list is basically a date that won't fall
# outside the defined partitions.

Time for a plain table = 98.1 sec

#part    parted    tg-direct-map    tg-if-else
=====    ======    =============    ==========
10       114.3     1483.3            742.4
50       112.5     1476.6           2016.8
100      117.1     1498.4           5386.1
500      125.3     1475.5             --
1000     129.9     1474.4             --
5000     137.5     1491.4             --
10000    154.7     1480.9             --


Then for a 2-level partitioned table with each of the above partitions
partitioned by list (c), with 10 sub-partitions each as follows:

ptab_NNNNN_a for values in ('a');
ptab_NNNNN_b for values in ('b');
...
ptab_NNNNN_k for values in ('j');

I didn't include the times for inheritance table with a routing trigger in
this case, as it seems that the results would look something like the above:

Time for a plain table = 98.1 sec

#part    (sub-)parted
=====    ============
10       127.0
50       152.3
100      156.6
500      191.8
1000     187.3


Regarding tuple-mapping-required vs no-tuple-mapping-required, all cases
currently require tuple-mapping, because the decision is based on the
result of comparing parent and partition TupleDesc using
equalTupleDescs(), which fails so quickly because TupleDesc.tdtypeid are
not the same.  Anyway, I simply commented out the tuple-mapping statement
in ExecInsert() to observe just slightly improved numbers as follows
(comparing with numbers in the table just above):

#part    (sub-)parted
=====    =================
10       113.9 (vs. 127.0)
100      135.7 (vs. 156.6)
500      182.1 (vs. 191.8)

Thanks,
Amit

On 2016/11/28 23:42, Ashutosh Bapat wrote:
> Here are some comments on 0003 patch.

Thanks for the review!

> 1. In ALTER TABLE documentation we should refer to CREATE TABLE documentation
> for partition_bound_spec syntax, similar ADD table_constraint note.

That makes sense, done.

>
> 2. I think, "of the target table" is missing after all the ... constraints
> +      match.  Also, it must have all the <literal>NOT NULL</literal> and
> +      <literal>CHECK</literal> constraints present in the target table.

The "present in the target table" part at the end of the sentence is
supposed to mean just that, but I changed it to say "of the target table"
instead, replacing "present in" by "of".

> 3. I think, using "any of the" instead of "some" is preferred in the following
> sentence. In the following sentence, we should use "such a constraint" instead
> of "constraints" to avoid mixed singular and plural usage.
> +      If some <literal>CHECK</literal> constraint of the table being attached

OK, done.

> 4. This construction is ambiguous. "are not considered" - considered where? for
> what? Constraints on which object?
> +      clause.  Currently <literal>UNIQUE</literal>, <literal>PRIMARY
> KEY</literal>,
> +      and <literal>FOREIGN KEY</literal> constraints are not considered.

That is essentially the same sentence as last sentence in the description
of INHERITS.  But I moved it next to: "Also, it must have all the NOT NULL
and CHECK constraints present in the target table."  Hopefully, that makes
it less ambiguous.  It's supposed to mean that those constraints are not
considered for inheritance unlike NOT NULL and CHECK constraints.

> 5. What is a partition constraint? Do we define that term anywhere in the
> documentation? If not we should define that term and then use it here.
> +      A full table scan is performed on the table being attached to check that
> +      no existing row in the table violates the partition constraint.  It is

Partition constraint is an implicit constraint derived from the parent's
partition key and its partition bounds enforced whenever an action is
applied to a partition directly - such as, inserting/updating a row and
when attaching a table as partition.  In fact, it also comes into play
during constraint exclusion when selecting from the parent table.

Do you think it's better to mention something like the above in the
description of CREATE TABLE PARTITION OF?

> 6. The paragraph following
> +      A full table scan is performed on the table
> being attached to check that seems to be confusing. It says that the
> potentially expensive scan can be avoided by adding a constraint equivalent to
> partition constraint. That seems to be wrong. The table will be scanned anyway,
> when adding a valid constraint per ALTER TABLE ADD table_constraint
> documentation. Instead you might want to say that the table scan will not be
> carried out if the existing constraints imply the partition constraints.

Ah, I see the confusion.  What's written is not supposed to mean that the
constraint be added after attaching the table as a partition, it's before.
 It's right that the table will be scanned anyway to validate the
constraint, but at that point, it's not related to the partitioned table
in any way and hence does not affect it.  As soon as the ATTACH PARTITION
command is executed, an exclusive lock will be acquired on the
parent/partitioned table, during which it's better to avoid any actions
that will take long - such as the validation scan.  If the constraint
added *beforehand* to the table being attached is such that it would allow
only a subset or a strict subset of the rows that the partition constraint
will allow, then it is not necessary to *repeat* the scan and hence is
not.  If it's not necessarily true that the constraints of the table being
attached will only allow a subset of a strict subset of the rows that the
partition constraint will allow, then the validation scan cannot be
avoided.  IOW, the advice is to help avoid the scan while taking an
exclusive lock on the parent.

Anyway, I have modified some sentences to be slightly clearer, see if that
works for you.

> 7. You might want to specify the fate of range or lists covered by the
> partition being detached in DETACH PARTITION section.

The partitioned table simply ceases to accept the rows falling in such a
range or a list.  The detached partition turn into a standalone table
which happens to contain only the rows with values of certain columns
(partition keys) within the range or the list.  However, there is no
explicit (catalogued) check constraint to that effect on the newly
standalone table, if that's what you were asking.

> 8. Since partition bound specification is more than a few words, you might want
> to refer to the actual syntax in CREATE TABLE command.
> +      <term><replaceable
> class="PARAMETER">partition_bound_spec</replaceable></term>
> +      <listitem>
> +       <para>
> +        The partition bound specification for a new partition.
> +       </para>
> +      </listitem>
> +     </varlistentry>

OK, added a link to the CREATE TABLE page.

>
> 9. I think, we need to use "may be scanned" instead of "is scanned", given that
> some other part of the documentation talks about "avoiding" the table scan. May
> be refer to that documentation to clarify the uncertainty.
> +    Similarly, when attaching a new partition it is scanned to verify that

Agreed, done.

> 10. The same paragraph talks about multiple ALTER TABLE commands being run
> together to avoid multiple table rewrites. Do we want to document whether
> ATTACH/DETACH partition can be run with any other command or not.

While currently ATTACH/DETACH can be run with any other ALTER TABLE
sub-command (much as ALTER TABLE INHERIT can), I think it's perhaps better
to prevent that.  For example, following awkward-sounding thing happens:

alter table p attach partition p1 for values in (1, 2, 3), add b int;
ERROR:  child table is missing column "b"

I have modified the patch so that the syntax does not allow such
combination of sub-commands.  Also, moved ATTACH PARTITION and DETACH
PARTITION in the synopsis section from "where action is one of:" area to
above where various forms of ALTER TABLE are listed.  Also specified down
below that ATTACH/DETACH PARTITION cannot be run in parallel with other
alterations.

> 11. ATTACH partition documentation is not talking about the unlogged or
> temporary tables being attached to is logged or non-temporary. What is current
> code doing about these cases? Do we allow such mixed partition hierarchy?

I tried to match the behavior of CREATE TABLE INHERITS() and ALTER TABLE
INHERIT

In ALTER TABLE child INHERIT parent case (as in CREATE TABLE child()
INHERITS(parent) case), failure occurs if the child is permanent rel and
parent temporary or if both are temporary, but if either parent or child
is of another session.  The same happens in case of ALTER TABLE parent
ATTACH PARTITION child and CREATE TABLE child PARTITION OF parent,
respectively.

LOGGED-ness is not considered here at all.


Attached updated patches.

Thanks,
Amit

On Fri, Nov 25, 2016 at 5:49 AM, Amit Langote
<Langote_Amit_f8@lab.ntt.co.jp> wrote:
> On 2016/11/25 11:44, Robert Haas wrote:
>> On Thu, Nov 24, 2016 at 6:13 AM, Amit Langote wrote:
>>> Also, it does nothing to help the undesirable situation that one can
>>> insert a row with a null partition key (expression) into any of the range
>>> partitions if targeted directly, because of how ExecQual() handles
>>> nullable constraint expressions (treats null value as satisfying the
>>> partition constraint).
>>
>> That's going to have to be fixed somehow.  How bad would it be if we
>> passed ExecQual's third argument as false for partition constraints?
>> Or else you could generate the actual constraint as expr IS NOT NULL
>> AND expr >= lb AND expr < ub.
>
> About the former, I think that might work.  If a column is NULL, it would
> be caught in ExecConstraints() even before ExecQual() is called, because
> of the NOT NULL constraint.  If an expression is NULL, or for some reason,
> the partitioning operator (=, >=, or <) returned NULL even for a non-NULL
> column or expression, then ExecQual() would fail if we passed false for
> resultForNull.  Not sure if that would be violating the SQL specification
> though.

I don't think the SQL specification can have anything to say about an
implicit constraint generated as an implementation detail of our
partitioning implementation.

> The latter would work too.  But I guess we would only emit expr IS NOT
> NULL, not column IS NOT NULL, because columns are covered by NOT NULL
> constraints.

Right.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Tue, Nov 29, 2016 at 6:24 AM, Amit Langote
<Langote_Amit_f8@lab.ntt.co.jp> wrote:
> # All times in seconds (on my modestly-powerful development VM)
> #
> # nrows = 10,000,000 generated using:
> #
> # INSERT INTO $tab
> # SELECT '$last'::date - ((s.id % $maxsecs + 1)::bigint || 's')::interval,
> #       (random() * 5000)::int % 4999 + 1,
> #        case s.id % 10
> #          when 0 then 'a'
> #          when 1 then 'b'
> #          when 2 then 'c'
> #          ...
> #          when 9 then 'j'
> #       end
> # FROM generate_series(1, $nrows) s(id)
> # ORDER BY random();
> #
> # The first item in the select list is basically a date that won't fall
> # outside the defined partitions.
>
> Time for a plain table = 98.1 sec
>
> #part    parted    tg-direct-map    tg-if-else
> =====    ======    =============    ==========
> 10       114.3     1483.3            742.4
> 50       112.5     1476.6           2016.8
> 100      117.1     1498.4           5386.1
> 500      125.3     1475.5             --
> 1000     129.9     1474.4             --
> 5000     137.5     1491.4             --
> 10000    154.7     1480.9             --

Very nice!

Obviously, it would be nice if the overhead were even lower, but it's
clearly a vast improvement over what we have today.

> Regarding tuple-mapping-required vs no-tuple-mapping-required, all cases
> currently require tuple-mapping, because the decision is based on the
> result of comparing parent and partition TupleDesc using
> equalTupleDescs(), which fails so quickly because TupleDesc.tdtypeid are
> not the same.  Anyway, I simply commented out the tuple-mapping statement
> in ExecInsert() to observe just slightly improved numbers as follows
> (comparing with numbers in the table just above):
>
> #part    (sub-)parted
> =====    =================
> 10       113.9 (vs. 127.0)
> 100      135.7 (vs. 156.6)
> 500      182.1 (vs. 191.8)

I think you should definitely try to get that additional speedup when
you can.  It doesn't seem like a lot when you think of how much is
already being saved, but a healthy number of users are going to
compare it to the performance on an unpartitioned table rather than to
our historical performance.   127/98.1 = 1.29, but 113.9/98.1 = 1.16
-- and obviously a 16% overhead from partitioning is way better than a
29% overhead, even if the old overhead was a million percent.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Thu, Dec 1, 2016 at 12:48 AM, Robert Haas <robertmhaas@gmail.com> wrote:
> On Fri, Nov 25, 2016 at 5:49 AM, Amit Langote wrote:
>> On 2016/11/25 11:44, Robert Haas wrote:
>>> On Thu, Nov 24, 2016 at 6:13 AM, Amit Langote wrote:
>>>> Also, it does nothing to help the undesirable situation that one can
>>>> insert a row with a null partition key (expression) into any of the range
>>>> partitions if targeted directly, because of how ExecQual() handles
>>>> nullable constraint expressions (treats null value as satisfying the
>>>> partition constraint).
>>>
>>> That's going to have to be fixed somehow.  How bad would it be if we
>>> passed ExecQual's third argument as false for partition constraints?
>>> Or else you could generate the actual constraint as expr IS NOT NULL
>>> AND expr >= lb AND expr < ub.
>>
>> About the former, I think that might work.  If a column is NULL, it would
>> be caught in ExecConstraints() even before ExecQual() is called, because
>> of the NOT NULL constraint.  If an expression is NULL, or for some reason,
>> the partitioning operator (=, >=, or <) returned NULL even for a non-NULL
>> column or expression, then ExecQual() would fail if we passed false for
>> resultForNull.  Not sure if that would be violating the SQL specification
>> though.
>
> I don't think the SQL specification can have anything to say about an
> implicit constraint generated as an implementation detail of our
> partitioning implementation.

Yeah, I thought so too.

>> The latter would work too.  But I guess we would only emit expr IS NOT
>> NULL, not column IS NOT NULL, because columns are covered by NOT NULL
>> constraints.
>
> Right.

The latest patch I posted earlier today has this implementation.

Thanks,
Amit

On Wed, Nov 30, 2016 at 10:56 AM, Amit Langote <amitlangote09@gmail.com> wrote:
> The latest patch I posted earlier today has this implementation.

I decided to try out these patches today with #define
CLOBBER_CACHE_ALWAYS 1 in pg_config_manual.h, which found a couple of
problems:

1. RelationClearRelation() wasn't preserving the rd_partkey, even
though there's plenty of code that relies on it not changing while we
hold a lock on the relation - in particular, transformPartitionBound.

2. partition_bounds_equal() was using the comparator and collation for
partitioning column 0 to compare the datums for all partitioning
columns.  It's amazing this passed the regression tests.

The attached incremental patch fixes those things and some cosmetic
issues I found along the way.

3. RelationGetPartitionDispatchInfo() is badly broken:

1010         pd[i] = (PartitionDispatch) palloc(sizeof(PartitionDispatchData));
1011         pd[i]->relid = RelationGetRelid(partrel);
1012         pd[i]->key = RelationGetPartitionKey(partrel);
1013         pd[i]->keystate = NIL;
1014         pd[i]->partdesc = partdesc;
1015         pd[i]->indexes = (int *) palloc(partdesc->nparts * sizeof(int));
1016         heap_close(partrel, NoLock);
1017
1018         m = 0;
1019         for (j = 0; j < partdesc->nparts; j++)
1020         {
1021             Oid     partrelid = partdesc->oids[j];

This code imagines that pointers it extracted from partrel are certain
to remain valid after heap_close(partrel, NoLock), perhaps on the
strength of the fact that we still retain a lock on the relation.  But
this isn't the case.  As soon as nobody has the relation open, a call
to RelationClearRelation() will destroy the relcache entry and
everything to which it points; with CLOBBER_CACHE_ALWAYS, I see a
failure at line 1021:

#0  RelationGetPartitionDispatchInfo (rel=0x1136dddf8, lockmode=3,
leaf_part_oids=0x7fff5633b938) at partition.c:1021
1021                Oid        partrelid = partdesc->oids[j];
(gdb) bt 5
#0  RelationGetPartitionDispatchInfo (rel=0x1136dddf8, lockmode=3,
leaf_part_oids=0x7fff5633b938) at partition.c:1021
#1  0x0000000109b8d71f in ExecInitModifyTable (node=0x7fd12984d750,
estate=0x7fd12b885438, eflags=0) at nodeModifyTable.c:1730
#2  0x0000000109b5e7ac in ExecInitNode (node=0x7fd12984d750,
estate=0x7fd12b885438, eflags=0) at execProcnode.c:159
#3  0x0000000109b58548 in InitPlan (queryDesc=0x7fd12b87b638,
eflags=0) at execMain.c:961
#4  0x0000000109b57dcd in standard_ExecutorStart
(queryDesc=0x7fd12b87b638, eflags=0) at execMain.c:239
(More stack frames follow...)
Current language:  auto; currently minimal
(gdb) p debug_query_string
$1 = 0x7fd12b84c238 "insert into list_parted values (null, 1);"
(gdb) p partdesc[0]
$2 = {
  nparts = 2139062143,
  oids = 0x7f7f7f7f7f7f7f7f,
  boundinfo = 0x7f7f7f7f7f7f7f7f
}

As you can see, the partdesc is no longer valid here.  I'm not
immediately sure how to fix this; this isn't a simple thinko.  You
need to keep the relations open for the whole duration of the query,
not just long enough to build the dispatch info.  I think you should
try to revise this so that each relation is opened once and kept open;
maybe the first loop should be making a pointer-list of Relations
rather than an int-list of relation OIDs.  And it seems to me (though
I'm getting a little fuzzy here because it's late) that you need all
of the partitions open, not just the ones that are subpartitioned,
because how else are you going to know how to remap the tuple if the
column order is different?  But I don't see this code doing that,
which makes me wonder if the partitions are being opened yet again in
some other location.

I recommend that once you fix this, you run 'make check' with #define
CLOBBER_CACHE_ALWAYS 1 and look for other hazards.  Such mistakes are
easy to make with this kind of patch.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 2016/12/07 13:38, Robert Haas wrote:
> On Wed, Nov 30, 2016 at 10:56 AM, Amit Langote <amitlangote09@gmail.com> wrote:
>> The latest patch I posted earlier today has this implementation.
>
> I decided to try out these patches today with #define
> CLOBBER_CACHE_ALWAYS 1 in pg_config_manual.h, which found a couple of
> problems:
>
> 1. RelationClearRelation() wasn't preserving the rd_partkey, even
> though there's plenty of code that relies on it not changing while we
> hold a lock on the relation - in particular, transformPartitionBound.

Oh, I thought an AccessExclusiveLock on the relation would prevent having
to worry about that, but guess I'm wrong.  Perhaps, having a lock on a
table does not preclude RelationClearRelation() resetting the table's
relcache.

> 2. partition_bounds_equal() was using the comparator and collation for
> partitioning column 0 to compare the datums for all partitioning
> columns.  It's amazing this passed the regression tests.

Oops, it seems that the regression tests where the above code might be
exercised consisted only of range partition key with columns all of the
same type: create table test(a int, b int) partition by range (a, (a+b));

> The attached incremental patch fixes those things and some cosmetic
> issues I found along the way.

Thanks for fixing these.  Attached patches include these changes.

> 3. RelationGetPartitionDispatchInfo() is badly broken:
>
> 1010         pd[i] = (PartitionDispatch) palloc(sizeof(PartitionDispatchData));
> 1011         pd[i]->relid = RelationGetRelid(partrel);
> 1012         pd[i]->key = RelationGetPartitionKey(partrel);
> 1013         pd[i]->keystate = NIL;
> 1014         pd[i]->partdesc = partdesc;
> 1015         pd[i]->indexes = (int *) palloc(partdesc->nparts * sizeof(int));
> 1016         heap_close(partrel, NoLock);
> 1017
> 1018         m = 0;
> 1019         for (j = 0; j < partdesc->nparts; j++)
> 1020         {
> 1021             Oid     partrelid = partdesc->oids[j];
>
> This code imagines that pointers it extracted from partrel are certain
> to remain valid after heap_close(partrel, NoLock), perhaps on the
> strength of the fact that we still retain a lock on the relation.  But
> this isn't the case.  As soon as nobody has the relation open, a call
> to RelationClearRelation() will destroy the relcache entry and
> everything to which it points; with CLOBBER_CACHE_ALWAYS, I see a
> failure at line 1021:
>
> #0  RelationGetPartitionDispatchInfo (rel=0x1136dddf8, lockmode=3,
> leaf_part_oids=0x7fff5633b938) at partition.c:1021
> 1021                Oid        partrelid = partdesc->oids[j];
> (gdb) bt 5
> #0  RelationGetPartitionDispatchInfo (rel=0x1136dddf8, lockmode=3,
> leaf_part_oids=0x7fff5633b938) at partition.c:1021
> #1  0x0000000109b8d71f in ExecInitModifyTable (node=0x7fd12984d750,
> estate=0x7fd12b885438, eflags=0) at nodeModifyTable.c:1730
> #2  0x0000000109b5e7ac in ExecInitNode (node=0x7fd12984d750,
> estate=0x7fd12b885438, eflags=0) at execProcnode.c:159
> #3  0x0000000109b58548 in InitPlan (queryDesc=0x7fd12b87b638,
> eflags=0) at execMain.c:961
> #4  0x0000000109b57dcd in standard_ExecutorStart
> (queryDesc=0x7fd12b87b638, eflags=0) at execMain.c:239
> (More stack frames follow...)
> Current language:  auto; currently minimal
> (gdb) p debug_query_string
> $1 = 0x7fd12b84c238 "insert into list_parted values (null, 1);"
> (gdb) p partdesc[0]
> $2 = {
>   nparts = 2139062143,
>   oids = 0x7f7f7f7f7f7f7f7f,
>   boundinfo = 0x7f7f7f7f7f7f7f7f
> }
>
> As you can see, the partdesc is no longer valid here.  I'm not
> immediately sure how to fix this; this isn't a simple thinko.  You
> need to keep the relations open for the whole duration of the query,
> not just long enough to build the dispatch info.  I think you should
> try to revise this so that each relation is opened once and kept open;
> maybe the first loop should be making a pointer-list of Relations
> rather than an int-list of relation OIDs.

Thanks for the explanation, I see the problem.  I changed
PartitionDispatchData such that its 1st field is now Relation (instead of
current Oid), where we keep the relation descriptor of the corresponding
partitioned table.  Currently, only the leaf relations are held open in
their respective ResultRelInfo's (ModifyTableState.mt_partitions) and
later closed in ExecEndModifyTable().  Similarly, we have the
ModifyTableState.mt_partition_dispatch_info array, each of whose members
holds open a partitioned relation using the new field.  We close that too
along with leaf partitions as just mentioned. Similarly in case of COPY FROM.

> And it seems to me (though
> I'm getting a little fuzzy here because it's late) that you need all
> of the partitions open, not just the ones that are subpartitioned,
> because how else are you going to know how to remap the tuple if the
> column order is different?  But I don't see this code doing that,
> which makes me wonder if the partitions are being opened yet again in
> some other location.

Actually leaf partitions are opened by the respective callers of
RelationGetPartitionDispatchInfo() and held open in the corresponding
ResultRelInfo's (ModifyTableState.mt_partitions).  As mentioned above,
they will be closed by either ExecEndModifyTable() or CopyFrom() after
finishing the INSERT or COPY, respectively.

> I recommend that once you fix this, you run 'make check' with #define
> CLOBBER_CACHE_ALWAYS 1 and look for other hazards.  Such mistakes are
> easy to make with this kind of patch.

With the attached latest version of the patches, I couldn't see any
failures with a CLOBBER_CACHE_ALWAYS build.

Thanks,
Amit

On Wed, Dec 7, 2016 at 6:42 AM, Amit Langote
<Langote_Amit_f8@lab.ntt.co.jp> wrote:
> On 2016/12/07 13:38, Robert Haas wrote:
>> On Wed, Nov 30, 2016 at 10:56 AM, Amit Langote <amitlangote09@gmail.com> wrote:
>>> The latest patch I posted earlier today has this implementation.
>>
>> I decided to try out these patches today with #define
>> CLOBBER_CACHE_ALWAYS 1 in pg_config_manual.h, which found a couple of
>> problems:
>>
>> 1. RelationClearRelation() wasn't preserving the rd_partkey, even
>> though there's plenty of code that relies on it not changing while we
>> hold a lock on the relation - in particular, transformPartitionBound.
>
> Oh, I thought an AccessExclusiveLock on the relation would prevent having
> to worry about that, but guess I'm wrong.  Perhaps, having a lock on a
> table does not preclude RelationClearRelation() resetting the table's
> relcache.

No, it sure doesn't.  The lock prevents the table from actually being
changed, so a reload will find data equivalent to what it had before,
but it doesn't prevent the backend's cache from being flushed.

>> 2. partition_bounds_equal() was using the comparator and collation for
>> partitioning column 0 to compare the datums for all partitioning
>> columns.  It's amazing this passed the regression tests.
>
> Oops, it seems that the regression tests where the above code might be
> exercised consisted only of range partition key with columns all of the
> same type: create table test(a int, b int) partition by range (a, (a+b));

It doesn't seem like it; you had this: create table part1 partition of
range_parted for values from ('a', 1) to ('a', 10);

>> I recommend that once you fix this, you run 'make check' with #define
>> CLOBBER_CACHE_ALWAYS 1 and look for other hazards.  Such mistakes are
>> easy to make with this kind of patch.
>
> With the attached latest version of the patches, I couldn't see any
> failures with a CLOBBER_CACHE_ALWAYS build.

Cool.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 2016-12-07 12:42, Amit Langote wrote:

> 0001-Catalog-and-DDL-for-partitioned-tables-20.patch
> 0002-psql-and-pg_dump-support-for-partitioned-tables-20.patch
> 0003-Catalog-and-DDL-for-partitions-20.patch
> 0004-psql-and-pg_dump-support-for-partitions-20.patch
> 0005-Teach-a-few-places-to-use-partition-check-quals-20.patch
> 0006-Tuple-routing-for-partitioned-tables-20.patch
> 0007-Update-DDL-Partitioning-chapter-to-reflect-new-devel-20.patch

Patches apply, compile, check OK.


But this yields a segfault:

begin;
create schema if not exists s;
create table s.t (c text, d text, id serial) partition by list 
((ascii(substring(coalesce(c, d, ''), 1, 1))));
create table s.t_part_ascii_065 partition of s.t for values in ( 65 );




it logs as follows:

2016-12-07 17:03:45.787 CET 6125 LOG:  server process (PID 11503) was 
terminated by signal 11: Segmentation fault
2016-12-07 17:03:45.787 CET 6125 DETAIL:  Failed process was running: 
create table s.t_part_ascii_065 partition of s.t for values in ( 65 );
2016-12-07 17:03:45.787 CET 6125 LOG:  terminating any other active 
server processes
2016-12-07 17:03:45.791 CET 6125 LOG:  all server processes terminated; 
reinitializing
2016-12-07 17:03:45.999 CET 11655 LOG:  database system was interrupted; 
last known up at 2016-12-07 17:00:38 CET
2016-12-07 17:03:48.040 CET 11655 LOG:  database system was not properly 
shut down; automatic recovery in progress
2016-12-07 17:03:48.156 CET 11655 LOG:  redo starts at 0/2897988
2016-12-07 17:03:48.172 CET 11655 LOG:  invalid magic number 0000 in log 
segment 000000010000000000000002, offset 9207808
2016-12-07 17:03:48.172 CET 11655 LOG:  redo done at 0/28C72C0
2016-12-07 17:03:48.172 CET 11655 LOG:  last completed transaction was 
at log time 2016-12-07 17:01:29.580562+01
2016-12-07 17:03:49.534 CET 11655 LOG:  MultiXact member wraparound 
protections are now enabled
2016-12-07 17:03:49.622 CET 6125 LOG:  database system is ready to 
accept connections



Thanks,

Erik Rijkers

Hi Erik,

On Thu, Dec 8, 2016 at 1:19 AM, Erik Rijkers <er@xs4all.nl> wrote:
> On 2016-12-07 12:42, Amit Langote wrote:
>
>> 0001-Catalog-and-DDL-for-partitioned-tables-20.patch
>> 0002-psql-and-pg_dump-support-for-partitioned-tables-20.patch
>> 0003-Catalog-and-DDL-for-partitions-20.patch
>> 0004-psql-and-pg_dump-support-for-partitions-20.patch
>> 0005-Teach-a-few-places-to-use-partition-check-quals-20.patch
>> 0006-Tuple-routing-for-partitioned-tables-20.patch
>> 0007-Update-DDL-Partitioning-chapter-to-reflect-new-devel-20.patch
>
>
> Patches apply, compile, check OK.

Thanks!

> But this yields a segfault:
>
> begin;
> create schema if not exists s;
> create table s.t (c text, d text, id serial) partition by list
> ((ascii(substring(coalesce(c, d, ''), 1, 1))));
> create table s.t_part_ascii_065 partition of s.t for values in ( 65 );
>
> it logs as follows:
>
> 2016-12-07 17:03:45.787 CET 6125 LOG:  server process (PID 11503) was
> terminated by signal 11: Segmentation fault
> 2016-12-07 17:03:45.787 CET 6125 DETAIL:  Failed process was running: create
> table s.t_part_ascii_065 partition of s.t for values in ( 65 );

Hm, will look into this in a few hours.

Thanks,
Amit

On Wed, Dec 7, 2016 at 11:34 AM, Amit Langote <amitlangote09@gmail.com> wrote:
>> begin;
>> create schema if not exists s;
>> create table s.t (c text, d text, id serial) partition by list
>> ((ascii(substring(coalesce(c, d, ''), 1, 1))));
>> create table s.t_part_ascii_065 partition of s.t for values in ( 65 );
>>
>> it logs as follows:
>>
>> 2016-12-07 17:03:45.787 CET 6125 LOG:  server process (PID 11503) was
>> terminated by signal 11: Segmentation fault
>> 2016-12-07 17:03:45.787 CET 6125 DETAIL:  Failed process was running: create
>> table s.t_part_ascii_065 partition of s.t for values in ( 65 );
>
> Hm, will look into this in a few hours.

My bad.  The fix I sent last night for one of the cache flush issues
wasn't quite right.  The attached seems to fix it.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 2016-12-07 17:38, Robert Haas wrote:
> On Wed, Dec 7, 2016 at 11:34 AM, Amit Langote <amitlangote09@gmail.com> 
> wrote:
>>> begin;
>>> create schema if not exists s;
>>> create table s.t (c text, d text, id serial) partition by list
>>> ((ascii(substring(coalesce(c, d, ''), 1, 1))));
>>> create table s.t_part_ascii_065 partition of s.t for values in ( 65 
>>> );
>>> 
>>> it logs as follows:
>>> 
>>> 2016-12-07 17:03:45.787 CET 6125 LOG:  server process (PID 11503) was
>>> terminated by signal 11: Segmentation fault
>>> 2016-12-07 17:03:45.787 CET 6125 DETAIL:  Failed process was running: 
>>> create
>>> table s.t_part_ascii_065 partition of s.t for values in ( 65 );
>> 
>> Hm, will look into this in a few hours.
> 
> My bad.  The fix I sent last night for one of the cache flush issues
> wasn't quite right.  The attached seems to fix it.

Yes, fixed here too.  Thanks.


> --
> Robert Haas
> EnterpriseDB: http://www.enterprisedb.com
> The Enterprise PostgreSQL Company

On Wed, Dec 7, 2016 at 11:53 AM, Erik Rijkers <er@xs4all.nl> wrote:
>> My bad.  The fix I sent last night for one of the cache flush issues
>> wasn't quite right.  The attached seems to fix it.
> Yes, fixed here too.  Thanks.

Thanks for the report - that was a good catch.

I've committed 0001 - 0006 with that correction and a few other
adjustments.  There's plenty of room for improvement here, and almost
certainly some straight-up bugs too, but I think we're at a point
where it will be easier and less error-prone to commit follow on
changes incrementally rather than by continuously re-reviewing a very
large patch set for increasingly smaller changes.

Some notes:

* We should try to teach the executor never to scan the parent.
That's never necessary with this system, and it might add significant
overhead.  We should also try to get rid of the idea of the parent
having storage (i.e. a relfilenode).

* The fact that, in some cases, locking requirements for partitioning
are stronger than those for inheritance is not good.  We made those
decisions for good reasons -- namely, data integrity and not crashing
the server -- but it would certainly be good to revisit those things
and see whether there's any room for improvement.

* I didn't commit 0007, which updates the documentation for this new
feature. That patch removes more lines than it adds, and I suspect
what is needed here
is an expansion of the documentation rather than a diminishing of it.

* The fact that there's no implementation of row movement should be
documented as a limitation.  We should also look at removing that
limitation.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 2016/12/08 1:53, Erik Rijkers wrote:
> On 2016-12-07 17:38, Robert Haas wrote:
>> On Wed, Dec 7, 2016 at 11:34 AM, Amit Langote <amitlangote09@gmail.com>
>> wrote:
>>>> begin;
>>>> create schema if not exists s;
>>>> create table s.t (c text, d text, id serial) partition by list
>>>> ((ascii(substring(coalesce(c, d, ''), 1, 1))));
>>>> create table s.t_part_ascii_065 partition of s.t for values in ( 65 );
>>>>
>>>> it logs as follows:
>>>>
>>>> 2016-12-07 17:03:45.787 CET 6125 LOG:  server process (PID 11503) was
>>>> terminated by signal 11: Segmentation fault
>>>> 2016-12-07 17:03:45.787 CET 6125 DETAIL:  Failed process was running:
>>>> create
>>>> table s.t_part_ascii_065 partition of s.t for values in ( 65 );
>>>
>>> Hm, will look into this in a few hours.
>>
>> My bad.  The fix I sent last night for one of the cache flush issues
>> wasn't quite right.  The attached seems to fix it.
> 
> Yes, fixed here too.  Thanks.

Thanks for reporting and the fix, Erik and Robert!

Thanks,
Amit

Hi Robert,

On 2016/12/08 3:20, Robert Haas wrote:
> On Wed, Dec 7, 2016 at 11:53 AM, Erik Rijkers <er@xs4all.nl> wrote:
>>> My bad.  The fix I sent last night for one of the cache flush issues
>>> wasn't quite right.  The attached seems to fix it.
>> Yes, fixed here too.  Thanks.
> 
> Thanks for the report - that was a good catch.
> 
> I've committed 0001 - 0006 with that correction and a few other
> adjustments.  There's plenty of room for improvement here, and almost
> certainly some straight-up bugs too, but I think we're at a point
> where it will be easier and less error-prone to commit follow on
> changes incrementally rather than by continuously re-reviewing a very
> large patch set for increasingly smaller changes.

+1 and thanks a lot for your and everyone else's very patient support in
reviewing the patches.

> Some notes:
> 
> * We should try to teach the executor never to scan the parent.
> That's never necessary with this system, and it might add significant
> overhead.  We should also try to get rid of the idea of the parent
> having storage (i.e. a relfilenode).

Agreed, I will start investigating.

> * The fact that, in some cases, locking requirements for partitioning
> are stronger than those for inheritance is not good.  We made those
> decisions for good reasons -- namely, data integrity and not crashing
> the server -- but it would certainly be good to revisit those things
> and see whether there's any room for improvement.

+1

> * I didn't commit 0007, which updates the documentation for this new
> feature. That patch removes more lines than it adds, and I suspect
> what is needed here
> is an expansion of the documentation rather than a diminishing of it.

Hmm, I had mixed feeling about what to do about that as well.  So now, we
have the description of various new features buried into VI. Reference
section of the documentation, which is simply meant as a command
reference.  I agree that the new partitioning warrants more expansion in
the DDL partitioning chapter.  Will see how that could be done.

> * The fact that there's no implementation of row movement should be
> documented as a limitation.  We should also look at removing that
> limitation.

Yes, something to improve.  By the way, since we currently mention INSERT
tuple-routing directly in the description of the partitioned tables in the
CREATE TABLE command reference, is that also the place to list this
particular limitation?  Or is UPDATE command reference rather the correct
place?

Thanks,
Amit

On 2016-12-07 13:20:04 -0500, Robert Haas wrote:
> On Wed, Dec 7, 2016 at 11:53 AM, Erik Rijkers <er@xs4all.nl> wrote:
> >> My bad.  The fix I sent last night for one of the cache flush issues
> >> wasn't quite right.  The attached seems to fix it.
> > Yes, fixed here too.  Thanks.
> 
> Thanks for the report - that was a good catch.

Congrats to everyone working on this! This is a large step forward.

- Andres

On Thu, Dec 8, 2016 at 1:39 PM, Andres Freund <andres@anarazel.de> wrote:
> On 2016-12-07 13:20:04 -0500, Robert Haas wrote:
>> On Wed, Dec 7, 2016 at 11:53 AM, Erik Rijkers <er@xs4all.nl> wrote:
>> >> My bad.  The fix I sent last night for one of the cache flush issues
>> >> wasn't quite right.  The attached seems to fix it.
>> > Yes, fixed here too.  Thanks.
>>
>> Thanks for the report - that was a good catch.
>
> Congrats to everyone working on this! This is a large step forward.

Congratulations to all! It was a long way to this result.
-- 
Michael

On 2016/12/08 3:20, Robert Haas wrote:
> On Wed, Dec 7, 2016 at 11:53 AM, Erik Rijkers <er@xs4all.nl> wrote:
>>> My bad.  The fix I sent last night for one of the cache flush issues
>>> wasn't quite right.  The attached seems to fix it.
>> Yes, fixed here too.  Thanks.
>
> Thanks for the report - that was a good catch.
>
> I've committed 0001 - 0006 with that correction and a few other
> adjustments.  There's plenty of room for improvement here, and almost
> certainly some straight-up bugs too, but I think we're at a point
> where it will be easier and less error-prone to commit follow on
> changes incrementally rather than by continuously re-reviewing a very
> large patch set for increasingly smaller changes.

Attached is a patch to fix some stale comments in the code and a minor
correction to one of the examples on the CREATE TABLE page.

Thanks,
Amit

Amit,

* Amit Langote (Langote_Amit_f8@lab.ntt.co.jp) wrote:
> Hmm, I had mixed feeling about what to do about that as well.  So now, we
> have the description of various new features buried into VI. Reference
> section of the documentation, which is simply meant as a command
> reference.  I agree that the new partitioning warrants more expansion in
> the DDL partitioning chapter.  Will see how that could be done.

Definitely.

> > * The fact that there's no implementation of row movement should be
> > documented as a limitation.  We should also look at removing that
> > limitation.
>
> Yes, something to improve.  By the way, since we currently mention INSERT
> tuple-routing directly in the description of the partitioned tables in the
> CREATE TABLE command reference, is that also the place to list this
> particular limitation?  Or is UPDATE command reference rather the correct
> place?

Both.

Thanks!

Stephen

On Wed, Dec 7, 2016 at 11:42 PM, Michael Paquier
<michael.paquier@gmail.com> wrote:
>> Congrats to everyone working on this! This is a large step forward.
>
> Congratulations to all! It was a long way to this result.

Yes.  The last effort in this area which I can remember was by Itagaki
Takahiro in 2010, so we've been waiting for this for more than 6
years.  It's really good that Amit was able to put in the effort to
produce a committable patch, and I think he deserves all of our thanks
for getting that done - and NTT deserves our thanks for paying him to
do it.

Even though I know he put in a lot more work than I did, let me just
say: phew, even reviewing that was a ton of work.

Of course, this is the beginning, not the end.  I've been thinking
about next steps -- here's an expanded list:

- more efficient plan-time partition pruning (constraint exclusion is too slow)
- run-time partition pruning
- partition-wise join (Ashutosh Bapat is already working on this)
- try to reduce lock levels
- hash partitioning
- the ability to create an index on the parent and have all of the
children inherit it; this should work something like constraint
inheritance.  you could argue that this doesn't add any real new
capability but it's a huge usability feature.
- teaching autovacuum enough about inheritance hierarchies for it to
update the parent statistics when they get stale despite the lack of
any actual inserts/updates/deletes to the parent.  this has been
pending for a long time, but it's only going to get more important
- row movement (aka avoiding the need for an ON UPDATE trigger on each
partition)
- insert (and eventually update) tuple routing for foreign partitions
- not scanning the parent
- fixing the insert routing so that we can skip tuple conversion where possible
- fleshing out the documentation

One thing I'm wondering is whether we can optimize away some of the
heavyweight locks.  For example, if somebody does SELECT * FROM ptab
WHERE id = 1, they really shouldn't need to lock the entire
partitioning hierarchy, but right now they do.  If the root knows
based on its own partitioning key that only one child is relevant, it
would be good to lock *only that child*.  For this feature to be
competitive, it needs to scale to at least a few thousand partitions,
and locking thousands of objects instead of one or two is bound to be
slow.  Similarly, you can imagine teaching COPY to lock partitions
only on demand; if no tuples are routed to a particular partition, we
don't need to lock it.  There's a manageability component here, too:
not locking partitions unnecessarily makes ti easier to get DDL on
other partitions through.  Alternatively, maybe we could rewrite the
lock manager to be hierarchical, so that you can take a single lock
that represents an AccessShareLock on all partitions and only need to
make one entry in the lock table to do it.  That means that attempts
to lock individual partitions need to check not only for a lock on
that partition but also on anything further up in the hierarchy, but
that might be a good trade if it gives us O(1) locking on the parent.
And maybe we could also have a level of the hierarchy that represents
every-table-in-the-database, for the benefit of pg_dump.  Of course,
rewriting the lock manager is a big project not for the faint of
heart, but I think if we don't it's going to be a scaling bottleneck.

We also need to consider other parts of the system that may not scale,
like pg_dump.  For a long time, we've been sorta-kinda willing to fix
the worst of the scalability problems with pg_dump, but that's really
no longer an adequate response.  People want 1000 partitions.  Heck,
people want 1,000,000 partitions, but getting to where 1000 partitions
works well would help PostgreSQL a lot.  Our oft-repeated line that
inheritance isn't designed for large numbers of inheritance children
is basically just telling people who have the use case where they need
that to go use some other product.  Partitioning, like replication, is
not an optional feature for a world-class database.  And, from a
technical point of view, I think we've now got an infrastructure that
really should be able to be scaled up considerably higher than what
we've been able to do in the past.  When we were stuck with
inheritance + constraint exclusion, we could say "well, there's not
really any point because you'll hit these other limits anyway".  But I
think now that's not really true.  This patch eliminates one of the
core scalability problems in this area, and provides infrastructure
for attacking some of the others.  I hope that people will step up and
do that.  There's a huge opportunity here for PostgreSQL to become
relevant in use cases where it currently falters badly, and we should
try to take advantage of it.  This patch is a big step by itself, but
if we ignore the potential to do more with this as the base we will be
leaving a lot of "win" on the table.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Hi everyone,

> Of course, this is the beginning, not the end.  I've been thinking
> about next steps -- here's an expanded list:

> - more efficient plan-time partition pruning (constraint
> exclusion is too slow)
> - run-time partition pruning
> - try to reduce lock levels
> ...

We (PostgresPro) have been working on pg_pathman for quite a while, and
since it's obviously going to become the thing of the past, it would be a
wasted effort if we didn't try to participate.

For starters, I'd love to work on both plan-time & run-time partition
pruning. I created a custom node for run-time partition elimination, so I
think I'm capable of developing something similar.


--
Dmitry Ivanov
Postgres Professional: http://www.postgrespro.com
Russian Postgres Company

On Thu, Dec 8, 2016 at 11:13 AM, Dmitry Ivanov <d.ivanov@postgrespro.ru> wrote:
> We (PostgresPro) have been working on pg_pathman for quite a while, and
> since it's obviously going to become the thing of the past, it would be a
> wasted effort if we didn't try to participate.
>
> For starters, I'd love to work on both plan-time & run-time partition
> pruning. I created a custom node for run-time partition elimination, so I
> think I'm capable of developing something similar.

That would be fantastic.  I and my colleagues at EnterpriseDB can
surely help review; of course, maybe you and some of your colleagues
would like to help review our patches, too.  Do you think this is
likely to be something where you can get something done quickly, with
the hope of getting it into v10?  Time is growing short, but it would
be great to polish this a little more before we ship it.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

> That would be fantastic.  I and my colleagues at EnterpriseDB can
> surely help review; of course, maybe you and some of your colleagues
> would like to help review our patches, too.

Certainly, I'll start reviewing as soon as I get familiar with the code.


> Do you think this is
> likely to be something where you can get something done quickly, with
> the hope of getting it into v10?

Yes, I've just cleared my schedule in order to make this possible. I'll
bring in the patches ASAP.


--
Dmitry Ivanov
Postgres Professional: http://www.postgrespro.com
Russian Postgres Company

On Thu, Dec 8, 2016 at 11:43 AM, Alexander Korotkov
<a.korotkov@postgrespro.ru> wrote:
> Great! And it is very cool that we have basic infrastructure already
> committed.  Thanks a lot to you and everybody involved.

Thanks.

>> of course, maybe you and some of your colleagues
>> would like to help review our patches, too.
> We understand our reviewing performance is not sufficient.  Will try to do
> better during next commitfest.

Not trying to throw stones, just want to get as much committed as
possible.  And I think our patches are good and valuable improvements
too, so I want to see them go in because they will help everybody.
Thanks for trying to increase the reviewing effort; it is sorely
needed.

>> Do you think this is
>> likely to be something where you can get something done quickly, with
>> the hope of getting it into v10?
>
> Yes, because we have set of features already implemented in pg_pathman.  In
> particular we have following features from your list and some more.
>
> - more efficient plan-time partition pruning (constraint exclusion is too
> slow)
> - run-time partition pruning
> - insert (and eventually update) tuple routing for foreign partitions
> - hash partitioning
> - not scanning the parent

That's a lot of stuff.  Getting even a couple of those would be a big win.

>> Time is growing short, but it would
>> be great to polish this a little more before we ship it.
>
> Yes. Getting at least some of this features committed to v10 would be great
> and improve partitioning usability a lot.

+1.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Thu, Dec 8, 2016 at 11:44 AM, Dmitry Ivanov <d.ivanov@postgrespro.ru> wrote:
>> That would be fantastic.  I and my colleagues at EnterpriseDB can
>> surely help review; of course, maybe you and some of your colleagues
>> would like to help review our patches, too.
>
> Certainly, I'll start reviewing as soon as I get familiar with the code.

Thanks!

>> Do you think this is
>> likely to be something where you can get something done quickly, with
>> the hope of getting it into v10?
>
> Yes, I've just cleared my schedule in order to make this possible. I'll
> bring in the patches ASAP.

Fantastic.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

From: pgsql-hackers-owner@postgresql.org
> [mailto:pgsql-hackers-owner@postgresql.org] On Behalf Of Alexander
> Korotkov
> Yes. Getting at least some of this features committed to v10 would be great
> and improve partitioning usability a lot.

I'm sorry for not contributing to the real partitioning feature, but I'm really looking forward to seeing the efficient
plan-timeand run-time partition pruning implemented in v10.  Recently, we failed to acquire a customer because they
couldnot achieve their performance goal due to the slow partition pruning compared to Oracle.  The batch app prepares a
SELECTstatement against a partitioned table, then executes it millions of time with different parameter values.  It
tooka long time to process Bind messages.
 

Another requirement was subpartitioning.  Will this be possible with the current infrastructure, or does this need
drasticchange?
 

Regards
Takayuki Tsunakawa

On 2016/12/09 10:09, Tsunakawa, Takayuki wrote:
> Another requirement was subpartitioning.  Will this be possible with the current infrastructure, or does this need
drasticchange?
 

It does support sub-partitioning, although the syntax is a bit different.

Thanks,
Amit

From: Amit Langote [mailto:Langote_Amit_f8@lab.ntt.co.jp]
> On 2016/12/09 10:09, Tsunakawa, Takayuki wrote:
> > Another requirement was subpartitioning.  Will this be possible with the
> current infrastructure, or does this need drastic change?
> 
> It does support sub-partitioning, although the syntax is a bit different.

Super great!  I'm excited to try the feature when I have time.  I hope I can contribute to the quality by find any
bug.

Regards
Takayuki Tsunakawa

Hi Stephen,

On 2016/12/08 22:35, Stephen Frost wrote:
>>> * The fact that there's no implementation of row movement should be
>>> documented as a limitation.  We should also look at removing that
>>> limitation.
>>
>> Yes, something to improve.  By the way, since we currently mention INSERT
>> tuple-routing directly in the description of the partitioned tables in the
>> CREATE TABLE command reference, is that also the place to list this
>> particular limitation?  Or is UPDATE command reference rather the correct
>> place?
> 
> Both.

Attached a documentation fix patch.

Actually, there was no mention on the INSERT reference page of
tuple-routing occurring in case of partitioned tables and also the
possibility of an error if a *partition* is directly targeted in an
INSERT. Mentioned that as well.

Thanks,
Amit

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 2016/12/09 0:25, Robert Haas wrote:
> On Wed, Dec 7, 2016 at 11:42 PM, Michael Paquier
> <michael.paquier@gmail.com> wrote:
>>> Congrats to everyone working on this! This is a large step forward.
>>
>> Congratulations to all! It was a long way to this result.
> 
> Yes.  The last effort in this area which I can remember was by Itagaki
> Takahiro in 2010, so we've been waiting for this for more than 6
> years.  It's really good that Amit was able to put in the effort to
> produce a committable patch, and I think he deserves all of our thanks
> for getting that done - and NTT deserves our thanks for paying him to
> do it.
> 
> Even though I know he put in a lot more work than I did, let me just
> say: phew, even reviewing that was a ton of work.

Absolutely!  Your review comments and design suggestions have been
instrumental in improving (and cutting down on the size of) the patches.

> Of course, this is the beginning, not the end.

+1000!

> I've been thinking
> about next steps -- here's an expanded list:
> 
> - more efficient plan-time partition pruning (constraint exclusion is too slow)
> - run-time partition pruning
> - partition-wise join (Ashutosh Bapat is already working on this)
> - try to reduce lock levels
> - hash partitioning
> - the ability to create an index on the parent and have all of the
> children inherit it; this should work something like constraint
> inheritance.  you could argue that this doesn't add any real new
> capability but it's a huge usability feature.
> - teaching autovacuum enough about inheritance hierarchies for it to
> update the parent statistics when they get stale despite the lack of
> any actual inserts/updates/deletes to the parent.  this has been
> pending for a long time, but it's only going to get more important
> - row movement (aka avoiding the need for an ON UPDATE trigger on each
> partition)
> - insert (and eventually update) tuple routing for foreign partitions
> - not scanning the parent
> - fixing the insert routing so that we can skip tuple conversion where possible
> - fleshing out the documentation

I would definitely want to contribute to some of these items.  It's great
that many others plan to contribute toward this as well.

> One thing I'm wondering is whether we can optimize away some of the
> heavyweight locks.  For example, if somebody does SELECT * FROM ptab
> WHERE id = 1, they really shouldn't need to lock the entire
> partitioning hierarchy, but right now they do.  If the root knows
> based on its own partitioning key that only one child is relevant, it
> would be good to lock *only that child*.  For this feature to be
> competitive, it needs to scale to at least a few thousand partitions,
> and locking thousands of objects instead of one or two is bound to be
> slow.  Similarly, you can imagine teaching COPY to lock partitions
> only on demand; if no tuples are routed to a particular partition, we
> don't need to lock it.  There's a manageability component here, too:
> not locking partitions unnecessarily makes ti easier to get DDL on
> other partitions through.  Alternatively, maybe we could rewrite the
> lock manager to be hierarchical, so that you can take a single lock
> that represents an AccessShareLock on all partitions and only need to
> make one entry in the lock table to do it.  That means that attempts
> to lock individual partitions need to check not only for a lock on
> that partition but also on anything further up in the hierarchy, but
> that might be a good trade if it gives us O(1) locking on the parent.
> And maybe we could also have a level of the hierarchy that represents
> every-table-in-the-database, for the benefit of pg_dump.  Of course,
> rewriting the lock manager is a big project not for the faint of
> heart, but I think if we don't it's going to be a scaling bottleneck.

Hierarchical lock manager stuff is interesting.  Are you perhaps alluding
to a new *intention* lock mode as described in the literature on multiple
granularity locking [1]?

> We also need to consider other parts of the system that may not scale,
> like pg_dump.  For a long time, we've been sorta-kinda willing to fix
> the worst of the scalability problems with pg_dump, but that's really
> no longer an adequate response.  People want 1000 partitions.  Heck,
> people want 1,000,000 partitions, but getting to where 1000 partitions
> works well would help PostgreSQL a lot.  Our oft-repeated line that
> inheritance isn't designed for large numbers of inheritance children
> is basically just telling people who have the use case where they need
> that to go use some other product.  Partitioning, like replication, is
> not an optional feature for a world-class database.  And, from a
> technical point of view, I think we've now got an infrastructure that
> really should be able to be scaled up considerably higher than what
> we've been able to do in the past.  When we were stuck with
> inheritance + constraint exclusion, we could say "well, there's not
> really any point because you'll hit these other limits anyway".  But I
> think now that's not really true.  This patch eliminates one of the
> core scalability problems in this area, and provides infrastructure
> for attacking some of the others.  I hope that people will step up and
> do that.  There's a huge opportunity here for PostgreSQL to become
> relevant in use cases where it currently falters badly, and we should
> try to take advantage of it.  This patch is a big step by itself, but
> if we ignore the potential to do more with this as the base we will be
> leaving a lot of "win" on the table.

Agreed on all counts.

Thanks,
Amit

[1] https://en.wikipedia.org/wiki/Multiple_granularity_locking

Hi, everyone!


08.12.16 18:25, Robert Haas wrote:
> Of course, this is the beginning, not the end.  I've been thinking
> about next steps -- here's an expanded list:
>
> - more efficient plan-time partition pruning (constraint exclusion is too slow)
> - run-time partition pruning
> - partition-wise join (Ashutosh Bapat is already working on this)
> - try to reduce lock levels
> - hash partitioning
> - the ability to create an index on the parent and have all of the
> children inherit it; this should work something like constraint
> inheritance.  you could argue that this doesn't add any real new
> capability but it's a huge usability feature.
> - teaching autovacuum enough about inheritance hierarchies for it to
> update the parent statistics when they get stale despite the lack of
> any actual inserts/updates/deletes to the parent.  this has been
> pending for a long time, but it's only going to get more important
> - row movement (aka avoiding the need for an ON UPDATE trigger on each
> partition)
> - insert (and eventually update) tuple routing for foreign partitions
> - not scanning the parent
> - fixing the insert routing so that we can skip tuple conversion where possible
> - fleshing out the documentation
>
I would like to work on two tasks: - insert (and eventually update) tuple routing for foreign partition. - the ability
tocreate an index on the parent and have all of the 
 
children inherit it;

The first one has been implemented in pg_pathman somehow, but the code 
relies on dirty hacks, so the FDW API has to be improved. As for the 
extended index support, it doesn't look like a super-hard task.

-- 
Maksim Milyutin
Postgres Professional: http://www.postgrespro.com
Russian Postgres Company

On Fri, Dec 9, 2016 at 3:16 PM, Venkata B Nagothi <nag1010@gmail.com> wrote:
> Hi,
>
> I am testing the partitioning feature from the latest master and got the
> following error while loading the data -
>
> db01=# create table orders_y1993 PARTITION OF orders FOR VALUES FROM
> ('1993-01-01') TO ('1993-12-31');
> CREATE TABLE
>
> db01=# copy orders from '/data/orders-1993.csv' delimiter '|';
> ERROR:  could not read block 6060 in file "base/16384/16412": read only 0 of
> 8192 bytes
> CONTEXT:  COPY orders, line 376589:
> "9876391|374509|O|54847|1997-07-16|3-MEDIUM       |Clerk#000001993|0|ithely
> regular pack"

Hmm.   Could you tell what relation the file/relfilenode 16412 belongs to?

Also, is orders_y1993 the only partition of orders?  How about \d+ orders?

Thanks,
Amit

Hi,

On 12/07/2016 07:20 PM, Robert Haas wrote:> On Wed, Dec 7, 2016 at 11:53 AM, Erik Rijkers <er@xs4all.nl> wrote:>>> My
bad. The fix I sent last night for one of the cache flush issues>>> wasn't quite right.  The attached seems to fix
it.>>Yes, fixed here too.  Thanks.>> Thanks for the report - that was a good catch.>> I've committed 0001 - 0006 with
thatcorrection and a few other> adjustments.  There's plenty of room for improvement here, and almost> certainly some
straight-upbugs too, but I think we're at a point> where it will be easier and less error-prone to commit follow on>
changesincrementally rather than by continuously re-reviewing a very> large patch set for increasingly smaller
changes.>

I've been working on a review / testing of the partitioning patch, but 
have been unable to submit it before the commit due to a lot of travel. 
However, at least some of the points seem to be still valid, so let me 
share it as an after-commit review. Most of the issues are fairly minor 
(possibly even nitpicking).


review
------

1) src/include/catalog/pg_partitioned_table.h contains this bit:
 * $PostgreSQL: pgsql/src/include/catalog/pg_partitioned_table.h $

2) I'm wondering whether having 'table' in the catalog name (and also in 
the new relkind) is too limiting. I assume we'll have partitioned 
indexes one day, for example - do we expect to use the same catalogs?

3) A comment within BeginCopy (in copy.c) says:
    * Initialize state for CopyFrom tuple routing.  Watch out for    * any foreign partitions.

But the code does not seem to be doing that (at least I don't see any 
obvious checks for foreign partitions). Also, the comment should 
probably at least mention why foreign partitions need extra care.

To nitpick, the 'pd' variable in that block seems rather pointless - we 
can assign directly to cstate->partition_dispatch_info.

4) I see GetIndexOpClass() got renamed to ResolveOpClass(). I find the 
new name rather strange - all other similar functions start with "Get", 
so I guess "GetOpClass()" would be better. But I wonder if the rename 
was even necessary, considering that it still deals with index operator 
classes (although now also in context of partition keys). If the rename 
really is needed, isn't that a sign that the function does not belong to 
indexcmds.c anymore?

5) Half the error messages use 'child table' while the other half uses 
'partition'. I think we should be using 'partition' as child tables 
really have little meaning outside inheritance (which is kinda hidden 
behind the new partitioning stuff).

6) The psql tab-completion seems a bit broken, because it only offers 
the partitions, not the parent table. Which is usually exactly the 
opposite of what the user wants.


testing
-------

I've also done quite a bit of testing with different partition layouts 
(single-level list/range partitioning, multi-level partitioning etc.), 
with fairly large number (~10k) of partitions. The scripts I used are 
available here: https://bitbucket.org/tvondra/partitioning-tests

1) There seems to be an issue when a partition is created and then 
accessed within the same transaction, i.e. for example

BEGIN;
... create parent ...
... create partition ....
... insert into parent ...
COMMIT;

which simply fails with an error like this:

ERROR:  no partition of relation "range_test_single" found for row
DETAIL:  Failing row contains (99000, 99000).

Of course, the partition is there. And interestingly enough, this works 
perfectly fine when executed without the explicit transaction, so I 
assume it's some sort of cache invalidation mix-up.

2) When doing a SELECT COUNT(*) from the partitioned table, I get a plan 
like this:
                                  QUERY PLAN
------------------------------------------------------------------- Finalize Aggregate  (cost=124523.64..124523.65
rows=1width=8)   ->  Gather  (cost=124523.53..124523.64 rows=1 width=8)         Workers Planned: 1         ->  Partial
Aggregate (cost=123523.53..123523.54 rows=1 ...)               ->  Append  (cost=0.00..108823.53 rows=5880001 width=0)
                  ->  Parallel Seq Scan on parent ...                     ->  Parallel Seq Scan on partition_1 ...
              ->  Parallel Seq Scan on partition_2 ...                     ->  Parallel Seq Scan on partition_3 ...
               ->  Parallel Seq Scan on partition_4 ...                     ->  Parallel Seq Scan on partition_5 ...
                ->  Parallel Seq Scan on partition_6 ...                     ... and the rest of the 10k partitions
 

So if I understand the plan correctly, we first do a parallel scan of 
the parent, then partition_1, partition_2 etc. But AFAIK we scan the 
tables in Append sequentially, and each partition only has 1000 rows 
each, making the parallel execution rather inefficient. Unless we scan 
the partitions concurrently.

In any case, as this happens even with plain inheritance, it's probably 
more about the parallel query than about the new partitioning patch.

3) The last issue I noticed is that while
    EXPLAIN SELECT * FROM partitioned_table WHERE id = 1292323;

works just fine (it takes fair amount of time to plan with 10k 
partitions, but that's expected), this
    EXPLAIN UPDATE partitioned_table SET x = 1 WHERE id = 1292323;

allocates a lot of memory (~8GB on my laptop, before it gets killed by 
OOM killer). Again, the same thing happens with plain inheritance-based 
partitioning, so it's probably not a bug in the partitioning patch.

I'm mentioning it here because I think the new partitioning will 
hopefully get more efficient and handle large partition counts more 
efficiently (the inheritance only really works for ~100 partitions, 
which is probably why no one complained about OOM during UPDATEs). Of 
course, 10k partitions is a bit extreme (good for testing, though).

regards

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Hi,

On 2016/12/11 10:02, Venkata B Nagothi wrote:
> On Fri, Dec 9, 2016 at 11:11 PM, Amit Langote <amitlangote09@gmail.com>
> wrote:
>> On Fri, Dec 9, 2016 at 3:16 PM, Venkata B Nagothi <nag1010@gmail.com>
>> wrote:
>>> I am testing the partitioning feature from the latest master and got the
>>> following error while loading the data -
>>>
>>> db01=# create table orders_y1993 PARTITION OF orders FOR VALUES FROM
>>> ('1993-01-01') TO ('1993-12-31');
>>> CREATE TABLE
>>>
>>> db01=# copy orders from '/data/orders-1993.csv' delimiter '|';
>>> ERROR:  could not read block 6060 in file "base/16384/16412": read only
>> 0 of
>>> 8192 bytes
>>> CONTEXT:  COPY orders, line 376589:
>>> "9876391|374509|O|54847|1997-07-16|3-MEDIUM
>>  |Clerk#000001993|0|ithely
>>> regular pack"
>>
>> Hmm.   Could you tell what relation the file/relfilenode 16412 belongs to?
>>
> 
> db01=# select relname from pg_class where relfilenode=16412 ;
>    relname
> --------------
>  orders_y1997
> (1 row)
> 
> 
> I VACUUMED the partition and then re-ran the copy command and no luck.
> 
> db01=# vacuum orders_y1997;
> VACUUM
> 
> db01=# copy orders from '/data/orders-1993.csv' delimiter '|';
> ERROR:  could not read block 6060 in file "base/16384/16412": read only 0
> of 8192 bytes
> CONTEXT:  COPY orders, line 376589:
> "9876391|374509|O|54847|1997-07-16|3-MEDIUM       |Clerk#000001993|0|ithely
> regular pack"
> 
> I do not quite understand the below behaviour as well. I VACUUMED 1997
> partition and then i got an error for 1992 partition and then after 1996
> and then after 1994 and so on.
> 

[ ... ]

> db01=# vacuum orders_y1997;
> VACUUM
> db01=# copy orders from '/data/orders-1993.csv' delimiter '|';
> ERROR:  could not read block 6060 in file "base/16384/16412": read only 0
> of 8192 bytes
> CONTEXT:  COPY orders, line 376589:
> "9876391|374509|O|54847|1997-07-16|3-MEDIUM       |Clerk#000001993|0|ithely
> regular pack"
> db01=#
> 
> Am i not understanding anything here ?

I could not reproduce this issue.  Also, I could not say what might have
gone wrong based only on the information I have seen so far.

Have you tried inserting the same data using insert?

create table orders_unpartitioned (like orders);
copy orders_unpartitioned from '/data/orders-1993.csv';
insert into orders select * from orders_unpartitioned;

Thanks,
Amit

Hi Tomas,

On 2016/12/12 10:02, Tomas Vondra wrote:
> On 12/07/2016 07:20 PM, Robert Haas wrote:
>> I've committed 0001 - 0006 with that correction and a few other
>> adjustments.  There's plenty of room for improvement here, and almost
>> certainly some straight-up bugs too, but I think we're at a point
>> where it will be easier and less error-prone to commit follow on
>> changes incrementally rather than by continuously re-reviewing a very
>> large patch set for increasingly smaller changes.
>>
> 
> I've been working on a review / testing of the partitioning patch, but
> have been unable to submit it before the commit due to a lot of travel.
> However, at least some of the points seem to be still valid, so let me
> share it as an after-commit review. Most of the issues are fairly minor
> (possibly even nitpicking).

Thanks a lot for the review comments and testing!

I attach a patch that implements some of the fixes that you suggest below.
 Also, I am merging a patch I sent earlier [1] to avoid having to look at
multiple patches.

> review
> ------
> 
> 1) src/include/catalog/pg_partitioned_table.h contains this bit:
> 
>  * $PostgreSQL: pgsql/src/include/catalog/pg_partitioned_table.h $

Fixed.

> 2) I'm wondering whether having 'table' in the catalog name (and also in
> the new relkind) is too limiting. I assume we'll have partitioned indexes
> one day, for example - do we expect to use the same catalogs?

I am not sure I understand your idea of partitioned indexes, but I doubt
it would require entries in the catalog under consideration.  Could you
perhaps elaborate more?

> 3) A comment within BeginCopy (in copy.c) says:
> 
>     * Initialize state for CopyFrom tuple routing.  Watch out for
>     * any foreign partitions.
> 
> But the code does not seem to be doing that (at least I don't see any
> obvious checks for foreign partitions). Also, the comment should probably
> at least mention why foreign partitions need extra care.

It's a stale comment after I took out the code that handled foreign
partitions in final versions of the tuple-routing patch.  You may have
noticed in the commit message that tuple-routing does not work for foreign
partitions, because the changes I'd proposed weren't that good.  Comment
fixed anyway.

> To nitpick, the 'pd' variable in that block seems rather pointless - we
> can assign directly to cstate->partition_dispatch_info.

OK, fixed.  Also, fixed similar code in ExecInitModifyTable().

> 4) I see GetIndexOpClass() got renamed to ResolveOpClass(). I find the new
> name rather strange - all other similar functions start with "Get", so I
> guess "GetOpClass()" would be better. But I wonder if the rename was even
> necessary, considering that it still deals with index operator classes
> (although now also in context of partition keys). If the rename really is
> needed, isn't that a sign that the function does not belong to indexcmds.c
> anymore?

The fact that both index keys and partition keys have very similar
specification syntax means there would be some common code handling the
same, of which this is one (maybe, only) example.  I didn't see much point
in finding a new place for the function, although I can see why it may be
a bit confusing to future readers of the code.

About the new name - seeing the top line in the old header comment, what
the function does is resolve a possibly explicitly specified operator
class name to an operator class OID.  I hadn't changed the name in my
original patch, but Robert suggested the rename, which I thought made sense.

> 5) Half the error messages use 'child table' while the other half uses
> 'partition'. I think we should be using 'partition' as child tables really
> have little meaning outside inheritance (which is kinda hidden behind the
> new partitioning stuff).

One way to go about it may be to check all sites that can possibly report
an error involving child tables (aka "partitions") whether they know from
the context which name to use.  I think it's possible, because we have
access to the parent relation in all such sites and looking at the relkind
can tell whether to call child tables "partitions".

> 6) The psql tab-completion seems a bit broken, because it only offers the
> partitions, not the parent table. Which is usually exactly the opposite of
> what the user wants.

Actually, I have not implemented any support for tab-completion for the
new DDL.  I will work on that.

> testing
> -------
> 
> I've also done quite a bit of testing with different partition layouts
> (single-level list/range partitioning, multi-level partitioning etc.),
> with fairly large number (~10k) of partitions. The scripts I used are
> available here: https://bitbucket.org/tvondra/partitioning-tests
> 
> 1) There seems to be an issue when a partition is created and then
> accessed within the same transaction, i.e. for example
> 
> BEGIN;
> ... create parent ...
> ... create partition ....
> ... insert into parent ...
> COMMIT;
> 
> which simply fails with an error like this:
> 
> ERROR:  no partition of relation "range_test_single" found for row
> DETAIL:  Failing row contains (99000, 99000).
> 
> Of course, the partition is there. And interestingly enough, this works
> perfectly fine when executed without the explicit transaction, so I assume
> it's some sort of cache invalidation mix-up.

That's a clearly bug.  A relcache invalidation on the parent was missing
in create table ... partition of code path.  Fixed.

> 2) When doing a SELECT COUNT(*) from the partitioned table, I get a plan
> like this:
> 
>                                   QUERY PLAN
> -------------------------------------------------------------------
>  Finalize Aggregate  (cost=124523.64..124523.65 rows=1 width=8)
>    ->  Gather  (cost=124523.53..124523.64 rows=1 width=8)
>          Workers Planned: 1
>          ->  Partial Aggregate  (cost=123523.53..123523.54 rows=1 ...)
>                ->  Append  (cost=0.00..108823.53 rows=5880001 width=0)
>                      ->  Parallel Seq Scan on parent ...
>                      ->  Parallel Seq Scan on partition_1 ...
>                      ->  Parallel Seq Scan on partition_2 ...
>                      ->  Parallel Seq Scan on partition_3 ...
>                      ->  Parallel Seq Scan on partition_4 ...
>                      ->  Parallel Seq Scan on partition_5 ...
>                      ->  Parallel Seq Scan on partition_6 ...
>                      ... and the rest of the 10k partitions
> 
> So if I understand the plan correctly, we first do a parallel scan of the
> parent, then partition_1, partition_2 etc. But AFAIK we scan the tables in
> Append sequentially, and each partition only has 1000 rows each, making
> the parallel execution rather inefficient. Unless we scan the partitions
> concurrently.
> 
> In any case, as this happens even with plain inheritance, it's probably
> more about the parallel query than about the new partitioning patch.

Yes, I have seen some discussion [2] about a Parallel Append, which would
result in plans you're probably thinking of.

> 3) The last issue I noticed is that while
> 
>     EXPLAIN SELECT * FROM partitioned_table WHERE id = 1292323;
> 
> works just fine (it takes fair amount of time to plan with 10k partitions,
> but that's expected), this
> 
>     EXPLAIN UPDATE partitioned_table SET x = 1 WHERE id = 1292323;
> 
> allocates a lot of memory (~8GB on my laptop, before it gets killed by OOM
> killer). Again, the same thing happens with plain inheritance-based
> partitioning, so it's probably not a bug in the partitioning patch.
> 
> I'm mentioning it here because I think the new partitioning will hopefully
> get more efficient and handle large partition counts more efficiently (the
> inheritance only really works for ~100 partitions, which is probably why
> no one complained about OOM during UPDATEs). Of course, 10k partitions is
> a bit extreme (good for testing, though).

Plans with the inheritance parents as target tables (update/delete) go
through inheritance_planner() in the optimizer, which currently has a
design that is based on certain assumptions about traditional inheritance.
 It's possible hopefully to make it less expensive for the partitioned tables.

Thanks,
Amit

[1]
https://www.postgresql.org/message-id/c9737515-36ba-6a42-f788-1b8867e2dd38%40lab.ntt.co.jp

[2]
https://www.postgresql.org/message-id/83755E07-C435-493F-9F93-F727604D66A1%40gmail.com

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 12/7/16 1:20 PM, Robert Haas wrote:
> I've committed 0001 - 0006 with that correction and a few other
> adjustments.  There's plenty of room for improvement here, and almost
> certainly some straight-up bugs too, but I think we're at a point
> where it will be easier and less error-prone to commit follow on
> changes incrementally rather than by continuously re-reviewing a very
> large patch set for increasingly smaller changes.

This page
<https://www.postgresql.org/docs/devel/static/ddl-partitioning.html>,
which is found via the index entry for "Partitioning", should be updated
for the new functionality.

-- 
Peter Eisentraut              http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On 12/12/2016 07:37 AM, Amit Langote wrote:
>
> Hi Tomas,
>
> On 2016/12/12 10:02, Tomas Vondra wrote:
>>
>> 2) I'm wondering whether having 'table' in the catalog name (and also in
>> the new relkind) is too limiting. I assume we'll have partitioned indexes
>> one day, for example - do we expect to use the same catalogs?
>
> I am not sure I understand your idea of partitioned indexes, but I doubt
> it would require entries in the catalog under consideration.  Could you
> perhaps elaborate more?
>

OK, let me elaborate. Let's say we have a partitioned table, and I want 
to create an index. The index may be either "global" i.e. creating a 
single relation for data from all the partitions, or "local" (i.e. 
partitioned the same way as the table).

Local indexes are easier to implement (it's essentially what we have 
now, except that we need to create the indexes manually for each 
partition), and don't work particularly well for some use cases (e.g. 
unique constraints). This is what I mean by "partitioned indexes".

If the index is partitioned just like the table, we probably won't need 
to copy the partition key info (so, nothing in pg_partitioned_table).
I'm not sure it makes sense to partition the index differently than the 
table - I don't see a case where that would be useful.

The global indexes would work better for the unique constraint use case, 
but it clearly contradicts our idea of TID (no information about which 
partition that references).

So maybe the catalog really only needs to track info about tables? Not 
sure. I'm just saying it'd be unfortunate to have _table in the name, 
and end up using it for indexes too.

>
>> 4) I see GetIndexOpClass() got renamed to ResolveOpClass(). I find the new
>> name rather strange - all other similar functions start with "Get", so I
>> guess "GetOpClass()" would be better. But I wonder if the rename was even
>> necessary, considering that it still deals with index operator classes
>> (although now also in context of partition keys). If the rename really is
>> needed, isn't that a sign that the function does not belong to indexcmds.c
>> anymore?
>
> The fact that both index keys and partition keys have very similar
> specification syntax means there would be some common code handling the
> same, of which this is one (maybe, only) example.  I didn't see much point
> in finding a new place for the function, although I can see why it may be
> a bit confusing to future readers of the code.
>
> About the new name - seeing the top line in the old header comment, what
> the function does is resolve a possibly explicitly specified operator
> class name to an operator class OID.  I hadn't changed the name in my
> original patch, but Robert suggested the rename, which I thought made sense.
>

OK, I don't particularly like the name but I can live with that.

>> 5) Half the error messages use 'child table' while the other half uses
>> 'partition'. I think we should be using 'partition' as child tables really
>> have little meaning outside inheritance (which is kinda hidden behind the
>> new partitioning stuff).
>
> One way to go about it may be to check all sites that can possibly report
> an error involving child tables (aka "partitions") whether they know from
> the context which name to use.  I think it's possible, because we have
> access to the parent relation in all such sites and looking at the relkind
> can tell whether to call child tables "partitions".
>

Clearly, this is a consequence of building the partitioning on top of 
inheritance (not objecting to that approach, merely stating a fact).

I'm fine with whatever makes the error messages more consistent, if it 
does not make the code significantly more complex. It's a bit confusing 
when some use 'child tables' and others 'partitions'. I suspect even a 
single DML command may return a mix of those, depending on where exactly 
it fails (old vs. new code).

>> 6) The psql tab-completion seems a bit broken, because it only offers the
>> partitions, not the parent table. Which is usually exactly the opposite of
>> what the user wants.
>
> Actually, I have not implemented any support for tab-completion for the
> new DDL.  I will work on that.
>

Aha! So the problem probably is that psql does not recognize the new 
'partitioned table' relkind, and only looks at regular tables.

>> testing
>> -------
>
>> 2) When doing a SELECT COUNT(*) from the partitioned table, I get a plan
>> like this:
>>
>>                                   QUERY PLAN
>> -------------------------------------------------------------------
>>  Finalize Aggregate  (cost=124523.64..124523.65 rows=1 width=8)
>>    ->  Gather  (cost=124523.53..124523.64 rows=1 width=8)
>>          Workers Planned: 1
>>          ->  Partial Aggregate  (cost=123523.53..123523.54 rows=1 ...)
>>                ->  Append  (cost=0.00..108823.53 rows=5880001 width=0)
>>                      ->  Parallel Seq Scan on parent ...
>>                      ->  Parallel Seq Scan on partition_1 ...
>>                      ->  Parallel Seq Scan on partition_2 ...
>>                      ->  Parallel Seq Scan on partition_3 ...
>>                      ->  Parallel Seq Scan on partition_4 ...
>>                      ->  Parallel Seq Scan on partition_5 ...
>>                      ->  Parallel Seq Scan on partition_6 ...
>>                      ... and the rest of the 10k partitions
>>
>> So if I understand the plan correctly, we first do a parallel scan of the
>> parent, then partition_1, partition_2 etc. But AFAIK we scan the tables in
>> Append sequentially, and each partition only has 1000 rows each, making
>> the parallel execution rather inefficient. Unless we scan the partitions
>> concurrently.
>>
>> In any case, as this happens even with plain inheritance, it's probably
>> more about the parallel query than about the new partitioning patch.
>
> Yes, I have seen some discussion [2] about a Parallel Append, which would
> result in plans you're probably thinking of.
>

Actually, I think that thread is about allowing partial paths even if 
only some appendrel members support it. My point is that in this case 
it's a bit silly to even build the partial paths, when the partitions 
only have 1000 rows each.

I kinda suspect we only look at the total appendrel rowcount estimate, 
but haven't checked.

>> 3) The last issue I noticed is that while
>>
>>     EXPLAIN SELECT * FROM partitioned_table WHERE id = 1292323;
>>
>> works just fine (it takes fair amount of time to plan with 10k partitions,
>> but that's expected), this
>>
>>     EXPLAIN UPDATE partitioned_table SET x = 1 WHERE id = 1292323;
>>
>> allocates a lot of memory (~8GB on my laptop, before it gets killed by OOM
>> killer). Again, the same thing happens with plain inheritance-based
>> partitioning, so it's probably not a bug in the partitioning patch.
>>
>> I'm mentioning it here because I think the new partitioning will hopefully
>> get more efficient and handle large partition counts more efficiently (the
>> inheritance only really works for ~100 partitions, which is probably why
>> no one complained about OOM during UPDATEs). Of course, 10k partitions is
>> a bit extreme (good for testing, though).
>
> Plans with the inheritance parents as target tables (update/delete)
> go through inheritance_planner() in the optimizer, which currently
> has a design that is based on certain assumptions about traditional
> inheritance. It's possible hopefully to make it less expensive for
> the partitioned  tables.
>

Yes, I know. I wasn't really reporting it as a bug in the partitioning 
patch, but more as a rather surprising difference between plain SELECT 
and UPDATE.

Am I right that one of the ambitions of the new partitioning is to 
improve behavior with large number of partitions?

At first I thought it's somewhat related to the FDW sharding (each node 
being a partition and having local subpartitions), but I realize the 
planner will only deal with the node partitions I guess.

regards

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Hi guys,

Looks like I've just encountered a bug. Please excuse me for the messy
email, I don't have much time at the moment.


Here's the test case:

create table test(val int) partition by range (val);
create table test_1 partition of test for values from (1) to (1000)
partition by range(val);
create table test_2 partition of test for values from (1000) to (2000)
partition by range(val);
create table test_1_1 partition of test_1 for values from (1) to (500)
partition by range(val);
create table test_1_2 partition of test_1 for values from (500) to (1000)
partition by range(val);
create table test_1_1_1 partition of test_1_1 for values from (1) to (500);
create table test_1_2_1 partition of test_1_2 for values from (500) to
(1000);


/* insert a row into "test_1_2_1" */
insert into test values(600);


/* what we EXPECT to see */
select *, tableoid::regclass from test;
 val |  tableoid
-----+------------
 600 | test_1_2_1
(1 row)


/* what we ACTUALLY see */
insert into test values(600);
ERROR:  no partition of relation "test_1_1" found for row
DETAIL:  Failing row contains (600).


How does this happen? This is how "PartitionDispatch" array looks like:

test | test_1 | test_2 | test_1_1 | test_1_2

which means that this code (partition.c : 1025):


/*
 * We can assign indexes this way because of the way
 * parted_rels has been generated.
 */
pd[i]->indexes[j] = -(i + 1 + m);


doesn't work, since partitions are not always placed right after the parent
(implied by index "m").

We have to take into account the total amount of partitions we've
encountered so far (right before index "i").

I've attached a patch with a hotfix, but the code looks so-so and has a
smell. I think it must be rewritten. This bug hunt surely took a while: I
had to recheck all of the steps several times.


--
Dmitry Ivanov
Postgres Professional: http://www.postgrespro.com
Russian Postgres Company
-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Huh, this code is broken as well. We have to ignore partitions that don't
have any subpartitions. Patch is attached below (v2).


--
Dmitry Ivanov
Postgres Professional: http://www.postgrespro.com
Russian Postgres Company
-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 2016/12/12 23:14, Peter Eisentraut wrote:
> On 12/7/16 1:20 PM, Robert Haas wrote:
>> I've committed 0001 - 0006 with that correction and a few other
>> adjustments.  There's plenty of room for improvement here, and almost
>> certainly some straight-up bugs too, but I think we're at a point
>> where it will be easier and less error-prone to commit follow on
>> changes incrementally rather than by continuously re-reviewing a very
>> large patch set for increasingly smaller changes.
> 
> This page
> <https://www.postgresql.org/docs/devel/static/ddl-partitioning.html>,
> which is found via the index entry for "Partitioning", should be updated
> for the new functionality.

A patch was included to do that, but it didn't do more than replacing the
old DDL listing by the new partitioning commands.  As Robert said in the
email you quoted, that's not enough.  So I'm trying to come up with a
patch updating the page explaining the new functionality better.

Thanks,
Amit

Hi,

On 2016/12/13 2:45, Dmitry Ivanov wrote:
> Huh, this code is broken as well. We have to ignore partitions that don't
> have any subpartitions. Patch is attached below (v2).

Good catch and thanks a lot for the patch!  I have revised it a bit and
added some explanatory comments to that function.

Attaching the above patch, along with some other patches posted earlier,
and one more patch fixing another bug I found.  Patch descriptions follow:

0001-Miscallaneous-code-and-comment-improvements.patch

Fixes some obsolete comments while improving others.  Also, implements
some of Tomas Vondra's review comments.

0002-Miscallaneous-documentation-improvements.patch

Fixes inconsistencies and improves some examples in the documentation.
Also, mentions the limitation regarding row movement.

0003-Invalidate-the-parent-s-relcache-after-partition-cre.patch

Fixes a bug reported by Tomas, whereby a parent's relcache was not
invalidated after creation of a new partition using CREATE TABLE PARTITION
OF.  This resulted in tuple-routing code not being to able to find a
partition that was created by the last command in a given transaction.

0004-Fix-a-bug-of-insertion-into-an-internal-partition.patch

Fixes a bug I found this morning, whereby an internal partition's
constraint would not be enforced if it is targeted directly.  See example
below:

create table p (a int, b char) partition by range (a);
create table p1 partition of p for values from (1) to (10) partition by
list (b);
create table p1a partition of p1 for values in ('a');
insert into p1 values (0, 'a');  -- should fail, but doesn't

0005-Fix-a-tuple-routing-bug-in-multi-level-partitioned-t.patch

Fixes a bug discovered by Dmitry Ivanov, whereby wrong indexes were
assigned to the partitions of lower levels (level > 1), causing spurious
"partition not found" error as demonstrated in his email [1].


Sorry about sending some of these patches repeatedly though.

Thanks,
Amit

[1]
https://www.postgresql.org/message-id/e6c56fe9-4b87-4f64-ac6f-bc99675f3f9e%40postgrespro.ru

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 2016/12/13 0:17, Tomas Vondra wrote:
> On 12/12/2016 07:37 AM, Amit Langote wrote:
>>
>> Hi Tomas,
>>
>> On 2016/12/12 10:02, Tomas Vondra wrote:
>>>
>>> 2) I'm wondering whether having 'table' in the catalog name (and also in
>>> the new relkind) is too limiting. I assume we'll have partitioned indexes
>>> one day, for example - do we expect to use the same catalogs?
>>
>> I am not sure I understand your idea of partitioned indexes, but I doubt
>> it would require entries in the catalog under consideration.  Could you
>> perhaps elaborate more?
>>
> 
> OK, let me elaborate. Let's say we have a partitioned table, and I want to
> create an index. The index may be either "global" i.e. creating a single
> relation for data from all the partitions, or "local" (i.e. partitioned
> the same way as the table).
> 
> Local indexes are easier to implement (it's essentially what we have now,
> except that we need to create the indexes manually for each partition),
> and don't work particularly well for some use cases (e.g. unique
> constraints). This is what I mean by "partitioned indexes".
> 
> If the index is partitioned just like the table, we probably won't need to
> copy the partition key info (so, nothing in pg_partitioned_table).
> I'm not sure it makes sense to partition the index differently than the
> table - I don't see a case where that would be useful.
> 
> The global indexes would work better for the unique constraint use case,
> but it clearly contradicts our idea of TID (no information about which
> partition that references).
> 
> So maybe the catalog really only needs to track info about tables? Not
> sure. I'm just saying it'd be unfortunate to have _table in the name, and
> end up using it for indexes too.

Hmm, I didn't quite think of the case where the index is partitioned
differently from the table, but perhaps that's possible with some other
databases.

What you describe as "local indexes" or "locally partitioned indexes" is
something I would like to see being pursued in the near term.  In that
case, we would allow defining indexes on the parent that are recursively
defined on the partitions and marked as inherited index, just like we have
inherited check constraints and NOT NULL constraints.  I have not studied
whether we could implement (globally) *unique* indexes with this scheme
though, wherein the index key is a superset of the partition key.

>>> 5) Half the error messages use 'child table' while the other half uses
>>> 'partition'. I think we should be using 'partition' as child tables really
>>> have little meaning outside inheritance (which is kinda hidden behind the
>>> new partitioning stuff).
>>
>> One way to go about it may be to check all sites that can possibly report
>> an error involving child tables (aka "partitions") whether they know from
>> the context which name to use.  I think it's possible, because we have
>> access to the parent relation in all such sites and looking at the relkind
>> can tell whether to call child tables "partitions".
>>
> 
> Clearly, this is a consequence of building the partitioning on top of
> inheritance (not objecting to that approach, merely stating a fact).
> 
> I'm fine with whatever makes the error messages more consistent, if it
> does not make the code significantly more complex. It's a bit confusing
> when some use 'child tables' and others 'partitions'. I suspect even a
> single DML command may return a mix of those, depending on where exactly
> it fails (old vs. new code).

So, we have mostly some old DDL (CREATE/ALTER TABLE) and maintenance
commands that understand inheritance.  All of the their error messages
apply to partitions as well, wherein they will be referred to as "child
tables" using old terms.  We now have some cases where the commands cause
additional error messages for only partitions because of additional
restrictions that apply to them.  We use "partitions" for them because
they are essentially new error messages.

There won't be a case where single DML command would mix the two terms,
because we do not allow mixing partitioning and regular inheritance.
Maybe I misunderstood you though.

>>> So if I understand the plan correctly, we first do a parallel scan of the
>>> parent, then partition_1, partition_2 etc. But AFAIK we scan the tables in
>>> Append sequentially, and each partition only has 1000 rows each, making
>>> the parallel execution rather inefficient. Unless we scan the partitions
>>> concurrently.
>>>
>>> In any case, as this happens even with plain inheritance, it's probably
>>> more about the parallel query than about the new partitioning patch.
>>
>> Yes, I have seen some discussion [2] about a Parallel Append, which would
>> result in plans you're probably thinking of.
>>
> 
> Actually, I think that thread is about allowing partial paths even if only
> some appendrel members support it. My point is that in this case it's a
> bit silly to even build the partial paths, when the partitions only have
> 1000 rows each.
> 
> I kinda suspect we only look at the total appendrel rowcount estimate, but
> haven't checked.

Oh, I misread.  Anyway, Parallel Append is something to think about.

>>> 3) The last issue I noticed is that while
>>>
>>>     EXPLAIN SELECT * FROM partitioned_table WHERE id = 1292323;
>>>
>>> works just fine (it takes fair amount of time to plan with 10k partitions,
>>> but that's expected), this
>>>
>>>     EXPLAIN UPDATE partitioned_table SET x = 1 WHERE id = 1292323;
>>>
>>> allocates a lot of memory (~8GB on my laptop, before it gets killed by OOM
>>> killer). Again, the same thing happens with plain inheritance-based
>>> partitioning, so it's probably not a bug in the partitioning patch.
>>>
>>> I'm mentioning it here because I think the new partitioning will hopefully
>>> get more efficient and handle large partition counts more efficiently (the
>>> inheritance only really works for ~100 partitions, which is probably why
>>> no one complained about OOM during UPDATEs). Of course, 10k partitions is
>>> a bit extreme (good for testing, though).
>>
>> Plans with the inheritance parents as target tables (update/delete)
>> go through inheritance_planner() in the optimizer, which currently
>> has a design that is based on certain assumptions about traditional
>> inheritance. It's possible hopefully to make it less expensive for
>> the partitioned  tables.
>>
> 
> Yes, I know. I wasn't really reporting it as a bug in the partitioning
> patch, but more as a rather surprising difference between plain SELECT and
> UPDATE.
> 
> Am I right that one of the ambitions of the new partitioning is to improve
> behavior with large number of partitions?

Yes.  Currently, SELECT planning is O(n) with significantly large constant
factor.  It is possible now to make it O(log n).  Also, if we can do away
with inheritance_planner() treatment for the *partitioned tables* in case
of UPDATE/DELETE, then that would be great.  That would mean their
planning time would be almost same as the SELECT case.

As you might know, we have volunteers to make this happen sooner [1], :)

> At first I thought it's somewhat related to the FDW sharding (each node
> being a partition and having local subpartitions), but I realize the
> planner will only deal with the node partitions I guess.

Yeah, planner would only have the local partitioning metadata at its
disposal.  Foreign tables can only be leaf partitions, which if need to be
scanned for a given query, will be scanned using a ForeignScan.

Thanks,
Amit

[1]
https://www.postgresql.org/message-id/426b2b01-61e0-43aa-bd84-c6fcf516f1c3%40postgrespro.ru

On Thu, Dec 8, 2016 at 11:58 PM, Amit Langote
<Langote_Amit_f8@lab.ntt.co.jp> wrote:
> Hierarchical lock manager stuff is interesting.  Are you perhaps alluding
> to a new *intention* lock mode as described in the literature on multiple
> granularity locking [1]?

Yes.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Fri, Dec 9, 2016 at 5:46 AM, Maksim Milyutin
<m.milyutin@postgrespro.ru> wrote:
> I would like to work on two tasks:
>  - insert (and eventually update) tuple routing for foreign partition.
>  - the ability to create an index on the parent and have all of the children
> inherit it;
>
> The first one has been implemented in pg_pathman somehow, but the code
> relies on dirty hacks, so the FDW API has to be improved. As for the
> extended index support, it doesn't look like a super-hard task.

Great!

I think that the second one will be fairly tricky.  You will need some
way of linking the child indexes back to the parent index.  And then
you will need to prohibit them from being dropped or modified
independent of the parent index.  And you will need to cascade ALTER
commands on the parent index (which will have no real storage, I hope)
down to the children.  Unfortunately, index names have to be unique on
a schema-wide basis, so we'll somehow need to generate names for the
"child" indexes.  But those names might collide with existing objects,
and will need to be preserved across a dump-and-restore.  The concept
is simple but getting all of the details right is hard.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Tue, Dec 13, 2016 at 1:58 AM, Amit Langote
<Langote_Amit_f8@lab.ntt.co.jp> wrote:
> Attaching the above patch, along with some other patches posted earlier,
> and one more patch fixing another bug I found.  Patch descriptions follow:
>
> 0001-Miscallaneous-code-and-comment-improvements.patch
>
> Fixes some obsolete comments while improving others.  Also, implements
> some of Tomas Vondra's review comments.

Committed with some pgindenting.

> 0002-Miscallaneous-documentation-improvements.patch
>
> Fixes inconsistencies and improves some examples in the documentation.
> Also, mentions the limitation regarding row movement.

Ignored because I committed what I think is the same or a similar
patch earlier.  Please resubmit any remaining changes.

> 0003-Invalidate-the-parent-s-relcache-after-partition-cre.patch
>
> Fixes a bug reported by Tomas, whereby a parent's relcache was not
> invalidated after creation of a new partition using CREATE TABLE PARTITION
> OF.  This resulted in tuple-routing code not being to able to find a
> partition that was created by the last command in a given transaction.

Shouldn't StorePartitionBound() be responsible for issuing its own
invalidations, as StorePartitionKey() already is?  Maybe you'd need to
pass "parent" as another argument, but that way you know you don't
have the same bug at some other place where the function is called.

> 0004-Fix-a-bug-of-insertion-into-an-internal-partition.patch
>
> Fixes a bug I found this morning, whereby an internal partition's
> constraint would not be enforced if it is targeted directly.  See example
> below:
>
> create table p (a int, b char) partition by range (a);
> create table p1 partition of p for values from (1) to (10) partition by
> list (b);
> create table p1a partition of p1 for values in ('a');
> insert into p1 values (0, 'a');  -- should fail, but doesn't

I expect I'm missing something here, but why do we need to hoist
RelationGetPartitionQual() out of InitResultRelInfo() instead of just
having BeginCopy() pass true instead of false?

(Also needs a rebase due to the pgindent cleanup.)

> 0005-Fix-a-tuple-routing-bug-in-multi-level-partitioned-t.patch
>
> Fixes a bug discovered by Dmitry Ivanov, whereby wrong indexes were
> assigned to the partitions of lower levels (level > 1), causing spurious
> "partition not found" error as demonstrated in his email [1].

Committed.  It might have been good to include a test case.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Hi hackers,

On 08.12.2016 19:44, Dmitry Ivanov wrote:
>> That would be fantastic.  I and my colleagues at EnterpriseDB can
>> surely help review; of course, maybe you and some of your colleagues
>> would like to help review our patches, too.
>
> Certainly, I'll start reviewing as soon as I get familiar with the code.
>
>
>> Do you think this is
>> likely to be something where you can get something done quickly, with
>> the hope of getting it into v10?
>
> Yes, I've just cleared my schedule in order to make this possible. I'll
> bring in the patches ASAP.
>
>

We've noticed that PartitionDispatch object is built on every INSERT 
query and that it could create unnecessary overhead. Wouldn't it be 
better to keep it in relcache?

Thanks!

-- 
Ildar Musin
i.musin@postgrespro.ru

On Tue, Dec 13, 2016 at 12:22 PM, Ildar Musin <i.musin@postgrespro.ru> wrote:
> We've noticed that PartitionDispatch object is built on every INSERT query
> and that it could create unnecessary overhead. Wouldn't it be better to keep
> it in relcache?

You might be able to cache some of that data in the relcache, but List
*keystate is pointing to query-lifespan data, so you can't cache that.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 2016/12/09 19:46, Maksim Milyutin wrote:
> I would like to work on two tasks:
>  - insert (and eventually update) tuple routing for foreign partition.
>  - the ability to create an index on the parent and have all of the
> children inherit it;
>
> The first one has been implemented in pg_pathman somehow, but the code
> relies on dirty hacks, so the FDW API has to be improved. As for the
> extended index support, it doesn't look like a super-hard task.

That would be great!  I'd like to help review the first one.

Best regards,
Etsuro Fujita

Hi,

On 13.12.2016 21:10, Robert Haas wrote:
> On Tue, Dec 13, 2016 at 12:22 PM, Ildar Musin <i.musin@postgrespro.ru> wrote:
>> We've noticed that PartitionDispatch object is built on every INSERT query
>> and that it could create unnecessary overhead. Wouldn't it be better to keep
>> it in relcache?
>
> You might be able to cache some of that data in the relcache, but List
> *keystate is pointing to query-lifespan data, so you can't cache that.
>

Yes, you are right. I meant mostly the 'indexes' field. I've measured 
insert performance with perf in case when there are thousand partitions 
and it seems that 34% of the time it takes to run 
RelationGetPartitionDispatchInfo() which builds this indexes array. And 
the most of the time it spends on acquiring locks on all partitions 
which is unnecessary if we're inserting in just a single partition. 
Probably we could avoid this by moving at least indexes field into cache.

-- 
Ildar Musin
i.musin@postgrespro.ru

Hi Amit,

On 12/13/2016 09:45 AM, Amit Langote wrote:
> On 2016/12/13 0:17, Tomas Vondra wrote:
>> On 12/12/2016 07:37 AM, Amit Langote wrote:
>>>
>>> Hi Tomas,
>>>
>>> On 2016/12/12 10:02, Tomas Vondra wrote:
>>>>
>>>> 2) I'm wondering whether having 'table' in the catalog name (and also in
>>>> the new relkind) is too limiting. I assume we'll have partitioned indexes
>>>> one day, for example - do we expect to use the same catalogs?
>>>
>>> I am not sure I understand your idea of partitioned indexes, but I doubt
>>> it would require entries in the catalog under consideration.  Could you
>>> perhaps elaborate more?
>>>
>>
>> OK, let me elaborate. Let's say we have a partitioned table, and I want to
>> create an index. The index may be either "global" i.e. creating a single
>> relation for data from all the partitions, or "local" (i.e. partitioned
>> the same way as the table).
>>
>> Local indexes are easier to implement (it's essentially what we have now,
>> except that we need to create the indexes manually for each partition),
>> and don't work particularly well for some use cases (e.g. unique
>> constraints). This is what I mean by "partitioned indexes".
>>
>> If the index is partitioned just like the table, we probably won't need to
>> copy the partition key info (so, nothing in pg_partitioned_table).
>> I'm not sure it makes sense to partition the index differently than the
>> table - I don't see a case where that would be useful.
>>
>> The global indexes would work better for the unique constraint use case,
>> but it clearly contradicts our idea of TID (no information about which
>> partition that references).
>>
>> So maybe the catalog really only needs to track info about tables? Not
>> sure. I'm just saying it'd be unfortunate to have _table in the name, and
>> end up using it for indexes too.
>
> Hmm, I didn't quite think of the case where the index is partitioned
> differently from the table, but perhaps that's possible with some other
> databases.
>

I haven't thought about that very deeply either, so perhaps it's an 
entirely silly idea. Also, probably quite complex to implement I guess, 
so unlikely to be pursued soon.

> What you describe as "local indexes" or "locally partitioned indexes" is
> something I would like to see being pursued in the near term.  In that
> case, we would allow defining indexes on the parent that are recursively
> defined on the partitions and marked as inherited index, just like we have
> inherited check constraints and NOT NULL constraints.  I have not studied
> whether we could implement (globally) *unique* indexes with this scheme
> though, wherein the index key is a superset of the partition key.
>

I think implementing UNIQUE constraint with local indexes is possible 
and possibly even fairly simple, but it likely requires SHARE lock on 
all partitions, which is not particularly nice.

When the partition key is referenced in the constraint, that may allow 
locking only a subset of the partitions, possibly even a single one. But 
with multi-level partitioning schemes that may be difficult.

Also, I don't think it's very likely to have the partitioning key as 
part of the unique constraint. For example 'users' table is unlikely to 
be distributed by 'login' and so on.

The  global indexes make this easier, because there's just a single 
index to check. But of course, attaching/detaching partitions gets more 
expensive.

Anyway, starting a detailed discussion about local/global indexes was 
not really what I meant to do.

>> Clearly, this is a consequence of building the partitioning on top of
>> inheritance (not objecting to that approach, merely stating a fact).
>>
>> I'm fine with whatever makes the error messages more consistent, if it
>> does not make the code significantly more complex. It's a bit confusing
>> when some use 'child tables' and others 'partitions'. I suspect even a
>> single DML command may return a mix of those, depending on where exactly
>> it fails (old vs. new code).
>
> So, we have mostly some old DDL (CREATE/ALTER TABLE) and maintenance
> commands that understand inheritance.  All of the their error messages
> apply to partitions as well, wherein they will be referred to as "child
> tables" using old terms.  We now have some cases where the commands cause
> additional error messages for only partitions because of additional
> restrictions that apply to them.  We use "partitions" for them because
> they are essentially new error messages.
>
> There won't be a case where single DML command would mix the two terms,
> because we do not allow mixing partitioning and regular inheritance.
> Maybe I misunderstood you though.
>

Don't we call inheritance-related functions from the new DDL? In that 
case we'd fail with 'child tables' error messages in the old code, and 
'partitions' in the new code. I'd be surprised if there was no such code 
reuse, but I haven't checked.

>> Am I right that one of the ambitions of the new partitioning is to improve
>> behavior with large number of partitions?
>
> Yes.  Currently, SELECT planning is O(n) with significantly large constant
> factor.  It is possible now to make it O(log n).  Also, if we can do away
> with inheritance_planner() treatment for the *partitioned tables* in case
> of UPDATE/DELETE, then that would be great.  That would mean their
> planning time would be almost same as the SELECT case.
>
> As you might know, we have volunteers to make this happen sooner [1], :)
>

Yes, I know. And it's great that you've managed to make the first step, 
getting all the infrastructure in, allowing others to build on that. 
Kudos to you!

>> At first I thought it's somewhat related to the FDW sharding (each node
>> being a partition and having local subpartitions), but I realize the
>> planner will only deal with the node partitions I guess.
>
> Yeah, planner would only have the local partitioning metadata at its
> disposal.  Foreign tables can only be leaf partitions, which if need to be
> scanned for a given query, will be scanned using a ForeignScan.
>

Right, makes sense. Still, I can imagine for example having many daily 
partitions and not having to merge them regularly just to reduce the 
number of partitions.

regards

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Hi everyone,

Looks like "sql_inheritance" GUC is affecting partitioned tables:

explain (costs off) select * from test;
          QUERY PLAN
------------------------------
 Append
   ->  Seq Scan on test
   ->  Seq Scan on test_1
   ->  Seq Scan on test_2
   ->  Seq Scan on test_1_1
   ->  Seq Scan on test_1_2
   ->  Seq Scan on test_1_1_1
   ->  Seq Scan on test_1_2_1
(8 rows)


set sql_inheritance = off;


explain (costs off) select * from test;
    QUERY PLAN
------------------
 Seq Scan on test
(1 row)


I might be wrong, but IMO this should not happen. Queries involving update,
delete etc on partitioned tables are basically broken. Moreover, there's no
point in performing such operations on a parent table that's supposed to be
empty at all times.

I've come up with a patch which fixes this behavior for UPDATE, DELETE,
TRUNCATE and also in transformTableEntry(). It might be hacky, but it gives
an idea.

I didn't touch RenameConstraint() and renameatt() since this would break
ALTER TABLE ONLY command.


--
Dmitry Ivanov
Postgres Professional: http://www.postgrespro.com
Russian Postgres Company
-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 2016/12/14 1:32, Robert Haas wrote:
> On Tue, Dec 13, 2016 at 1:58 AM, Amit Langote
> <Langote_Amit_f8@lab.ntt.co.jp> wrote:
>> Attaching the above patch, along with some other patches posted earlier,
>> and one more patch fixing another bug I found.  Patch descriptions follow:
>>
>> 0001-Miscallaneous-code-and-comment-improvements.patch
>>
>> Fixes some obsolete comments while improving others.  Also, implements
>> some of Tomas Vondra's review comments.
> 
> Committed with some pgindenting.

Thanks!

>> 0002-Miscallaneous-documentation-improvements.patch
>>
>> Fixes inconsistencies and improves some examples in the documentation.
>> Also, mentions the limitation regarding row movement.
> 
> Ignored because I committed what I think is the same or a similar
> patch earlier.  Please resubmit any remaining changes.

It seems this patch is almost the same thing as what you committed.

>> 0003-Invalidate-the-parent-s-relcache-after-partition-cre.patch
>>
>> Fixes a bug reported by Tomas, whereby a parent's relcache was not
>> invalidated after creation of a new partition using CREATE TABLE PARTITION
>> OF.  This resulted in tuple-routing code not being to able to find a
>> partition that was created by the last command in a given transaction.
> 
> Shouldn't StorePartitionBound() be responsible for issuing its own
> invalidations, as StorePartitionKey() already is?  Maybe you'd need to
> pass "parent" as another argument, but that way you know you don't
> have the same bug at some other place where the function is called.

OK, done that way in PATCH 1/7 (of the attached various patches as
described below).

>> 0004-Fix-a-bug-of-insertion-into-an-internal-partition.patch
>>
>> Fixes a bug I found this morning, whereby an internal partition's
>> constraint would not be enforced if it is targeted directly.  See example
>> below:
>>
>> create table p (a int, b char) partition by range (a);
>> create table p1 partition of p for values from (1) to (10) partition by
>> list (b);
>> create table p1a partition of p1 for values in ('a');
>> insert into p1 values (0, 'a');  -- should fail, but doesn't
> 
> I expect I'm missing something here, but why do we need to hoist
> RelationGetPartitionQual() out of InitResultRelInfo() instead of just
> having BeginCopy() pass true instead of false?
> 
> (Also needs a rebase due to the pgindent cleanup.)

In this case, we want to enforce only the main target relation's partition
constraint (only needed if it happens to be an internal node partition),
not leaf partitions', because the latter is unnecessary.

We do InitResultRelInfo() for every leaf partition.  What
RelationGetPartitionQual() would return to InitResultRelInfo() is the
partition constraint of the individual leaf partitions, which as just
mentioned is unnecessary, and also would be inefficient.  With the
proposed patch, we only retrieve the partition constraint for the targeted
table (if there is any) once.  However, when assigning to
ri_PartitionCheck of individual leaf partition's ResultRelInfo, we still
must map any Vars in the expression from the target tables's attnos to the
corresponding leaf partition's attnos (previous version of the patch
failed to do that).

>> 0005-Fix-a-tuple-routing-bug-in-multi-level-partitioned-t.patch
>>
>> Fixes a bug discovered by Dmitry Ivanov, whereby wrong indexes were
>> assigned to the partitions of lower levels (level > 1), causing spurious
>> "partition not found" error as demonstrated in his email [1].
>
> Committed.  It might have been good to include a test case.

Agreed, added tests in the attached patch PATCH 7/7.


Aside from the above, I found few other issues and fixed them in the
attached patches.  Descriptions follow:

[PATCH 1/7] Invalidate the parent's relcache after partition creation.

Invalidate parent's relcache after a partition is created using CREATE
TABLE PARTITION OF.  (Independent reports by Keith Fiske and David Fetter)

[PATCH 2/7] Change how RelationGetPartitionQual() and related code works

Since we always want to recurse, ie, include the parent's partition
constraint (if any), get rid of the argument recurse.

Refactor out the code doing the mapping of attnos of Vars in partition
constraint expressions (parent attnos -> child attnos).  Move it to a
separate function map_partition_varattnos() and call it from appropriate
places.  It previously used to be done in get_qual_from_partbound(),
which would lead to wrong results in certain multi-level partitioning
cases, as the mapping would be done for immediate parent-partition pairs.
Now in generate_partition_qual() which is the workhorse of
RelationGetPartitionQual(), we first generate the full expression
(considering all levels of partitioning) and then do the mapping from the
root parent to a leaf partition.  It is also possible to generate
partition constraint up to certain non-leaf level and then apply the
same to leaf partitions of that sub-tree after suitable substitution
of varattnos using the new map_partition_varattnos() directly.

Bug fix: ATExecAttachPartition() failed to do the mapping when attaching
a partitioned table as partition. It is possible for the partitions of
such table to have different attributes from the table being attached
and/or the target partitioned table.

[PATCH 3/7] Refactor tuple-routing setup code

It's unnecessarily repeated in copy.c and nodeModifyTable.c, which makes
it a burden to maintain.  Should've been like this to begin with.

I moved the common code to ExecSetupPartitionTupleRouting() in execMain.c
that also houses ExecFindParttion() currently.  Hmm, should there be a
new src/backend/executor/execPartition.c?

[PATCH 4/7] Fix a bug of insertion into an internal partition.

Since implicit partition constraints are not inherited, an internal
partition's constraint was not being enforced when targeted directly.
So, include such constraint when setting up leaf partition result
relations for tuple-routing.

InitResultRelInfo()'s API changes with this.  Instead of passing
a boolean telling whether or not to load the partition constraint,
callers now need to pass the exact constraint expression to use
as ri_PartitionCheck or NIL.

[PATCH 5/7] Fix oddities of tuple-routing and TupleTableSlots

We must use the partition's tuple descriptor *after* a tuple is routed,
not the root table's.  Partition's attributes, for example, may be
ordered diferently from the root table's.

We must then switch back to the root table's for the next tuple and
so on.  A dedicated TupleTableSlot is allocated within EState called
es_partition_tuple_slot whose descriptor is set to a given leaf
partition for every row after it's routed.

[PATCH 6/7] Make ExecConstraints() emit the correct row in error msgs.

After a tuple is routed to a partition, it has been converted from the
root table's rowtype to the partition's.  If such a tuple causes an
error in ExecConstraints(), the row shown in error messages might not
match the input row due to possible differences between the root
table's rowtype and the partition's.

To convert back to the correct row format, keep root table relation
descriptor and a reverse tuple conversion map in the ResultRelInfo's
of leaf partitions.

[PATCH 7/7] Add some tests for recent fixes to PartitionDispatch code in
a25665088d

Thanks,
Amit

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Just poking around with partitioning. I notice that "\d parent"
doesn't list all the partitions, suggesting to use \d+ but a plain
"\d" does indeed list the partitions. That seems a bit strange and
also probably impractical if you have hundreds or thousands of
partitions. Has this come up in previous discussions? Unfortunately
it's proving a bit hard to search for "\d" :/

On 2016/12/16 17:02, Amit Langote wrote:
> [PATCH 2/7] Change how RelationGetPartitionQual() and related code works
> 
> Since we always want to recurse, ie, include the parent's partition
> constraint (if any), get rid of the argument recurse.
> 
> Refactor out the code doing the mapping of attnos of Vars in partition
> constraint expressions (parent attnos -> child attnos).  Move it to a
> separate function map_partition_varattnos() and call it from appropriate
> places.  It previously used to be done in get_qual_from_partbound(),
> which would lead to wrong results in certain multi-level partitioning
> cases, as the mapping would be done for immediate parent-partition pairs.
> Now in generate_partition_qual() which is the workhorse of
> RelationGetPartitionQual(), we first generate the full expression
> (considering all levels of partitioning) and then do the mapping from the
> root parent to a leaf partition.  It is also possible to generate
> partition constraint up to certain non-leaf level and then apply the
> same to leaf partitions of that sub-tree after suitable substitution
> of varattnos using the new map_partition_varattnos() directly.
> 
> Bug fix: ATExecAttachPartition() failed to do the mapping when attaching
> a partitioned table as partition. It is possible for the partitions of
> such table to have different attributes from the table being attached
> and/or the target partitioned table.

Oops, PATCH 2/7 attached with the previous email had a bug in it, whereby
map_partition_varattnos() was not applied to the partition constraint
expressions returned directly from the relcache (rd_partcheck) copy.
Attaching just the updated (only) PATCH 2 which fixes that.  Patches
1,3,4,5,6,7/7 from the previous email [1] are fine.  Sorry about that.

Thanks,
Amit

[1]
https://www.postgresql.org/message-id/c820c0eb-6935-6f84-8c6a-785fdff130c1@lab.ntt.co.jp

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 2016/12/16 17:38, Greg Stark wrote:
> Just poking around with partitioning. I notice that "\d parent"
> doesn't list all the partitions, suggesting to use \d+ but a plain
> "\d" does indeed list the partitions. That seems a bit strange and
> also probably impractical if you have hundreds or thousands of
> partitions. Has this come up in previous discussions? Unfortunately
> it's proving a bit hard to search for "\d" :/

Do you mean a plain "\d" (without an argument) should not list tables that
are partitions?  I think that might be preferable.  That would mean, we
list only the root partitioned tables with a plain "\d".

Regarding "\d parent", it does the same thing as regular inheritance, but
using the term "partition" instead of "child table".  Without specifying a
+ (\d parent), one gets just "Number of partitions: # (Use \d+ to list
them.)" and with + (\d+ parent), one gets the full list of partitions
showing the partition bound with each.

Thanks,
Amit

Hi Dmitry,

On 2016/12/16 0:40, Dmitry Ivanov wrote:
> Hi everyone,
> 
> Looks like "sql_inheritance" GUC is affecting partitioned tables:
> 
> explain (costs off) select * from test;
>          QUERY PLAN          ------------------------------
> Append
>   ->  Seq Scan on test
>   ->  Seq Scan on test_1
>   ->  Seq Scan on test_2
>   ->  Seq Scan on test_1_1
>   ->  Seq Scan on test_1_2
>   ->  Seq Scan on test_1_1_1
>   ->  Seq Scan on test_1_2_1
> (8 rows)
> 
> 
> set sql_inheritance = off;
> 
> 
> explain (costs off) select * from test;
>    QUERY PLAN    ------------------
> Seq Scan on test
> (1 row)
> 
> 
> I might be wrong, but IMO this should not happen. Queries involving
> update, delete etc on partitioned tables are basically broken. Moreover,
> there's no point in performing such operations on a parent table that's
> supposed to be empty at all times.
> 
> I've come up with a patch which fixes this behavior for UPDATE, DELETE,
> TRUNCATE and also in transformTableEntry(). It might be hacky, but it
> gives an idea.
> 
> I didn't touch RenameConstraint() and renameatt() since this would break
> ALTER TABLE ONLY command.

@@ -1198,6 +1198,12 @@ ExecuteTruncate(TruncateStmt *stmt)        rels = lappend(rels, rel);        relids =
lappend_oid(relids,myrelid);
 

+        /* Use interpretInhOption() unless it's a partitioned table */
+        if (rel->rd_rel->relkind != RELKIND_PARTITIONED_TABLE)
+            recurse = interpretInhOption(rv->inhOpt);
+        else
+            recurse = true;
+        if (recurse)        {            ListCell   *child;

If you see the else block of this if, you'll notice this:

else if (rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)           ereport(ERROR,
(errcode(ERRCODE_WRONG_OBJECT_TYPE),                   errmsg("must truncate child tables too")));
 

So that you get this behavior:

# set sql_inheritance to off;
SET

# truncate p;
ERROR:  must truncate child tables too

# reset sql_inheritance;
RESET

# truncate only p;
ERROR:  must truncate child tables too

# truncate p;
TRUNCATE TABLE

Beside that, I initially had implemented the same thing as what you are
proposing here, but reverted to existing behavior at some point during the
discussion. I think the idea behind was to not *silently* ignore user
specified configuration and instead error out with appropriate message.
While it seems to work reasonably for DDL and maintenance commands (like
TRUNCATE above), things sound strange for SELECT/UPDATE/DELETE as you're
saying.

Thanks,
Amit

On Fri, Dec 16, 2016 at 3:02 AM, Amit Langote
<Langote_Amit_f8@lab.ntt.co.jp> wrote:
> Aside from the above, I found few other issues and fixed them in the
> attached patches.  Descriptions follow:

To avoid any further mistakes on my part, can you please resubmit
these with each patch file containing a proposed commit message
including patch authorship information, who reported the issue, links
to relevant discussion if any, and any other attribution information
which I should not fail to include when committing?

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Sat, Dec 17, 2016 at 1:07 AM, Robert Haas <robertmhaas@gmail.com> wrote:
> On Fri, Dec 16, 2016 at 3:02 AM, Amit Langote
> <Langote_Amit_f8@lab.ntt.co.jp> wrote:
>> Aside from the above, I found few other issues and fixed them in the
>> attached patches.  Descriptions follow:
>
> To avoid any further mistakes on my part, can you please resubmit
> these with each patch file containing a proposed commit message
> including patch authorship information, who reported the issue, links
> to relevant discussion if any, and any other attribution information
> which I should not fail to include when committing?

I think it's a good advice and will keep in mind for any patches I
post henceforth.

In this particular case, I found all the issues myself while working
with some more esoteric test scenarios, except the first patch (1/7),
where I have mentioned in the description of the patch in the email,
that there were independent reports of the issue by Tomas Vondra and
David Fetter.

Thanks,
Amit

On 2016/12/17 11:32, Amit Langote wrote:
> On Sat, Dec 17, 2016 at 1:07 AM, Robert Haas <robertmhaas@gmail.com> wrote:
>> On Fri, Dec 16, 2016 at 3:02 AM, Amit Langote
>> <Langote_Amit_f8@lab.ntt.co.jp> wrote:
>>> Aside from the above, I found few other issues and fixed them in the
>>> attached patches.  Descriptions follow:
>>
>> To avoid any further mistakes on my part, can you please resubmit
>> these with each patch file containing a proposed commit message
>> including patch authorship information, who reported the issue, links
>> to relevant discussion if any, and any other attribution information
>> which I should not fail to include when committing?
> 
> I think it's a good advice and will keep in mind for any patches I
> post henceforth.
> 
> In this particular case, I found all the issues myself while working
> with some more esoteric test scenarios, except the first patch (1/7),
> where I have mentioned in the description of the patch in the email,
> that there were independent reports of the issue by Tomas Vondra and
> David Fetter.

Here are updated patches including the additional information.

Thanks,
Amit

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Sun, Dec 18, 2016 at 10:00 PM, Amit Langote
<Langote_Amit_f8@lab.ntt.co.jp> wrote:
> Here are updated patches including the additional information.

Thanks.  Committed 0001.  Will have to review the others when I'm less tired.

BTW, elog() is only supposed to be used for can't happen error
messages; when it is used, no translation is done.  So this is wrong:
   if (skip_validate)       elog(NOTICE, "skipping scan to validate partition constraint");


-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 2016/12/20 12:59, Robert Haas wrote:
> On Sun, Dec 18, 2016 at 10:00 PM, Amit Langote
> <Langote_Amit_f8@lab.ntt.co.jp> wrote:
>> Here are updated patches including the additional information.
> 
> Thanks.  Committed 0001.  Will have to review the others when I'm less tired.

Thanks!

> BTW, elog() is only supposed to be used for can't happen error
> messages; when it is used, no translation is done.  So this is wrong:
> 
>     if (skip_validate)
>         elog(NOTICE, "skipping scan to validate partition constraint");

You're right.  I was using it for debugging when I first wrote that code,
but then thought it would be better to eventually turn that into
ereport(INFO/NOTICE) for the final submission, which I missed to do. Sorry
about that.  Here's a patch.

Thanks,
Amit

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Amit Langote wrote:

> diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
> index 1c219b03dd..6a179596ce 100644
> --- a/src/backend/commands/tablecmds.c
> +++ b/src/backend/commands/tablecmds.c
> @@ -13297,8 +13297,10 @@ ATExecAttachPartition(List **wqueue, Relation rel, PartitionCmd *cmd)
>          }
>      }
>  
> +    /* It's safe to skip the validation scan after all */
>      if (skip_validate)
> -        elog(NOTICE, "skipping scan to validate partition constraint");
> +        ereport(INFO,
> +                (errmsg("skipping scan to validate partition constraint")));

Why not just remove the message altogether?

-- 
Álvaro Herrera                https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On Tue, Dec 20, 2016 at 4:51 AM, Alvaro Herrera
<alvherre@2ndquadrant.com> wrote:
> Amit Langote wrote:
>
>> diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
>> index 1c219b03dd..6a179596ce 100644
>> --- a/src/backend/commands/tablecmds.c
>> +++ b/src/backend/commands/tablecmds.c
>> @@ -13297,8 +13297,10 @@ ATExecAttachPartition(List **wqueue, Relation rel, PartitionCmd *cmd)
>>               }
>>       }
>>
>> +     /* It's safe to skip the validation scan after all */
>>       if (skip_validate)
>> -             elog(NOTICE, "skipping scan to validate partition constraint");
>> +             ereport(INFO,
>> +                             (errmsg("skipping scan to validate partition constraint")));
>
> Why not just remove the message altogether?

That's certainly an option.  It might be noise in some situations.  On
the other hand, it affects whether attaching the partition is O(1) or
O(n), so somebody might well want to know.  Or maybe they might be
more likely to want a message in the reverse situation, telling them
that the partition constraint DOES need to be validated.  I'm not sure
what the best user interface is here; thoughts welcome.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Robert Haas wrote:
> On Tue, Dec 20, 2016 at 4:51 AM, Alvaro Herrera
> <alvherre@2ndquadrant.com> wrote:
> > Amit Langote wrote:
> >
> >> diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
> >> index 1c219b03dd..6a179596ce 100644
> >> --- a/src/backend/commands/tablecmds.c
> >> +++ b/src/backend/commands/tablecmds.c
> >> @@ -13297,8 +13297,10 @@ ATExecAttachPartition(List **wqueue, Relation rel, PartitionCmd *cmd)
> >>               }
> >>       }
> >>
> >> +     /* It's safe to skip the validation scan after all */
> >>       if (skip_validate)
> >> -             elog(NOTICE, "skipping scan to validate partition constraint");
> >> +             ereport(INFO,
> >> +                             (errmsg("skipping scan to validate partition constraint")));
> >
> > Why not just remove the message altogether?
> 
> That's certainly an option.  It might be noise in some situations.  On
> the other hand, it affects whether attaching the partition is O(1) or
> O(n), so somebody might well want to know.  Or maybe they might be
> more likely to want a message in the reverse situation, telling them
> that the partition constraint DOES need to be validated.  I'm not sure
> what the best user interface is here; thoughts welcome.

Even if we decide to keep the message, I think it's not very good
wording anyhow; as a translator I disliked it on sight.  Instead of
"skipping scan to validate" I would use "skipping validation scan",
except that it's not clear what it is we're validating.  Mentioning
partition constraint in errcontext() doesn't like a great solution, but
I can't think of anything better.

(We have the table_rewrite event trigger, for a very similar use case.)

-- 
Álvaro Herrera                https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On Tue, Dec 20, 2016 at 10:27 AM, Alvaro Herrera
<alvherre@2ndquadrant.com> wrote:
> Even if we decide to keep the message, I think it's not very good
> wording anyhow; as a translator I disliked it on sight.  Instead of
> "skipping scan to validate" I would use "skipping validation scan",
> except that it's not clear what it is we're validating.  Mentioning
> partition constraint in errcontext() doesn't like a great solution, but
> I can't think of anything better.

Maybe something like: partition constraint for table \"%s\" is implied
by existing constraints

> (We have the table_rewrite event trigger, for a very similar use case.)

Hmm, maybe we should see if we can safely support an event trigger here, too.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Robert Haas wrote:
> On Tue, Dec 20, 2016 at 10:27 AM, Alvaro Herrera
> <alvherre@2ndquadrant.com> wrote:
> > Even if we decide to keep the message, I think it's not very good
> > wording anyhow; as a translator I disliked it on sight.  Instead of
> > "skipping scan to validate" I would use "skipping validation scan",
> > except that it's not clear what it is we're validating.  Mentioning
> > partition constraint in errcontext() doesn't like a great solution, but
> > I can't think of anything better.
> 
> Maybe something like: partition constraint for table \"%s\" is implied
> by existing constraints

Actually, shouldn't we emit a message if we *don't* skip the check?


-- 
Álvaro Herrera                https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On Mon, Dec 19, 2016 at 10:59 PM, Robert Haas <robertmhaas@gmail.com> wrote:
> On Sun, Dec 18, 2016 at 10:00 PM, Amit Langote
> <Langote_Amit_f8@lab.ntt.co.jp> wrote:
>> Here are updated patches including the additional information.
>
> Thanks.  Committed 0001.  Will have to review the others when I'm less tired.

0002. Can you add a test case for the bug fixed by this patch?

0003. Loses equalTupleDescs() check and various cases where
ExecOpenIndexes can be skipped.  Isn't that bad?  Also, "We locked all
the partitions above including the leaf partitions" should say "Caller
must have locked all the partitions including the leaf partitions".

0004. Unnecessary whitespace change in executor.h.  Still don't
understand why we need to hoist RelationGetPartitionQual() into the
caller.

0005. Can you add a test case for the bug fixed by this patch?

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 2016/12/21 1:53, Robert Haas wrote:
> On Mon, Dec 19, 2016 at 10:59 PM, Robert Haas <robertmhaas@gmail.com> wrote:
>> On Sun, Dec 18, 2016 at 10:00 PM, Amit Langote
>> <Langote_Amit_f8@lab.ntt.co.jp> wrote:
>>> Here are updated patches including the additional information.
>>
>> Thanks.  Committed 0001.  Will have to review the others when I'm less tired.
> 
> 0002. Can you add a test case for the bug fixed by this patch?

Done (also see below).

> 0003. Loses equalTupleDescs() check and various cases where
> ExecOpenIndexes can be skipped.  Isn't that bad?

I realized that as far as checking whether tuple conversion mapping is
required, the checks performed by convert_tuples_by_name() at the
beginning of the function following the comment /* Verify compatibility
and prepare attribute-number map */ are enough.  If equalTupleDescs()
returned false (which it always does because tdtypeid are never the same
for passed in tuple descriptors), we would be repeating some of the tests
in convert_tuples_by_name() anyway.

As for the checks performed for ExecOpenIndices(), it seems would be
better to keep the following in place, so added back:

        if (leaf_part_rri->ri_RelationDesc->rd_rel->relhasindex &&
            leaf_part_rri->ri_IndexRelationDescs == NULL)
            ExecOpenIndices(leaf_part_rri, false);

> Also, "We locked all
> the partitions above including the leaf partitions" should say "Caller
> must have locked all the partitions including the leaf partitions".

No, we do the locking in RelationGetPartitionDispatchInfo(), which is
called by ExecSetupPartitionTupleRouting() itself.

In ExecSetupPartitionTupleRouting() is the first time we lock all the
partitions.

> 0004. Unnecessary whitespace change in executor.h.  Still don't
> understand why we need to hoist RelationGetPartitionQual() into the
> caller.

We only need to check a result relation's (ri_RelationDesc's) partition
constraint if we are inserting into the result relation directly.  In case
of tuple-routing, we do not want to check the leaf partitions' partition
constraint, but if the direct target in that case is an internal
partition, we must check its partition constraint, which is same for all
leaf partitions in that sub-tree.  It wouldn't be wrong per se to check
each leaf partition's constraint in that case, which includes the target
partitioned table's constraint as well, but that would inefficient due to
both having to retrieve the constraints and having ExecConstraints()
*unnecessarily* check it for every row.

If we keep doing RelationGetPartitionQual() in InitResultRelInfo()
controlled by a bool argument (load_partition_check), we cannot avoid the
above mentioned inefficiency if we want to fix this bug.

> 0005. Can you add a test case for the bug fixed by this patch?

Done, but...

Breaking changes into multiple commits/patches does not seem to work for
adding regression tests.  So, I've combined multiple patches into a single
patch which is now patch 0002 in the attached set of patches.  Its commit
message is very long now.  To show an example of bugs that 0002 is meant for:

create table p (a int, b int) partition by range (a, b);
create table p1 (b int, a int not null) partition by range (b);
create table p11 (like p1);
alter table p11 drop a;
alter table p11 add a int;
alter table p11 drop a;
alter table p11 add a int not null;

# select attrelid::regclass, attname, attnum
  from pg_attribute
  where attnum > 0
  and (attrelid = 'p'::regclass
    or attrelid = 'p1'::regclass
    or attrelid = 'p11'::regclass) and attname = 'a';
 attrelid | attname | attnum
----------+---------+--------
 p        | a       |      1
 p1       | a       |      2
 p11      | a       |      4
(3 rows)

alter table p1 attach partition p11 for values from (1) to (5);
alter table p attach partition p1 for values from (1, 1) to (1, 10);

-- the following is wrong
# insert into p11 (a, b) values (10, 4);
INSERT 0 1

-- wrong too (using the wrong TupleDesc after tuple routing)
# insert into p1 (a, b) values (10, 4);
ERROR:  null value in column "a" violates not-null constraint
DETAIL:  Failing row contains (4, null).

-- once we fix the wrong TupleDesc issue
# insert into p1 (a, b) values (10, 4);
INSERT 0 1

which is wrong because p1, as a partition of p, should not accept 10 for
a.  But its partition constraint is not being applied to the leaf
partition p11 into which the tuple is routed (the bug).

Thanks,
Amit

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Wed, Dec 21, 2016 at 5:33 AM, Amit Langote
<Langote_Amit_f8@lab.ntt.co.jp> wrote:
> Breaking changes into multiple commits/patches does not seem to work for
> adding regression tests.  So, I've combined multiple patches into a single
> patch which is now patch 0002 in the attached set of patches.

Ugh, seriously?  It's fine to combine closely related bug fixes but
not all of these are.  I don't see why you can't add some regression
tests in one patch and then add some more in the next patch.

Meanwhile, committed the latest 0001 and the elog() patch.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 2016/12/22 1:50, Robert Haas wrote:
> On Wed, Dec 21, 2016 at 5:33 AM, Amit Langote
> <Langote_Amit_f8@lab.ntt.co.jp> wrote:
>> Breaking changes into multiple commits/patches does not seem to work for
>> adding regression tests.  So, I've combined multiple patches into a single
>> patch which is now patch 0002 in the attached set of patches.
> 
> Ugh, seriously?  It's fine to combine closely related bug fixes but
> not all of these are.  I don't see why you can't add some regression
> tests in one patch and then add some more in the next patch.

I managed to do that this time around with the attached set of patches.
Guess I gave up too easily in the previous attempt.

While working on that, I discovered yet-another-bug having to do with the
tuple descriptor that's used as we route a tuple down a partition tree. If
attnums of given key attribute(s) are different on different levels, it
would be incorrect to use the original slot's (one passed by ExecInsert())
tuple descriptor to inspect the original slot's heap tuple, as we go down
the tree.  It might cause spurious "partition not found" at some level due
to looking at incorrect field in the input tuple because of using the
wrong tuple descriptor (root table's attnums not always same as other
partitioned tables in the tree).  Patch 0001 fixes that including a test.
It also addresses the problem I mentioned previously that once
tuple-routing is done, we failed to switch to a slot with the leaf
partition's tupdesc (IOW, continued to use the original slot with root
table's tupdesc causing spurious failures due to differences in attums
between the leaf partition and the root table).

Further patches 0002, 0003 and 0004 fix bugs that I sent one-big-patch for
in my previous message.  Each patch has a test for the bug it's meant to fix.

Patch 0005 is the same old "Add some more tests for tuple-routing" per [1]:

> Meanwhile, committed the latest 0001 and the elog() patch.

Thanks!

Regards,
Amit

[1]
https://www.postgresql.org/message-id/CA%2BTgmoZ86v1G%2Bzx9etMiSQaBBvYMKfU-iitqZArSh5z0n8Q4cA%40mail.gmail.com

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Thu, Dec 22, 2016 at 3:35 AM, Amit Langote
<Langote_Amit_f8@lab.ntt.co.jp> wrote:
> While working on that, I discovered yet-another-bug having to do with the
> tuple descriptor that's used as we route a tuple down a partition tree. If
> attnums of given key attribute(s) are different on different levels, it
> would be incorrect to use the original slot's (one passed by ExecInsert())
> tuple descriptor to inspect the original slot's heap tuple, as we go down
> the tree.  It might cause spurious "partition not found" at some level due
> to looking at incorrect field in the input tuple because of using the
> wrong tuple descriptor (root table's attnums not always same as other
> partitioned tables in the tree).  Patch 0001 fixes that including a test.

I committed this, but I'm a bit uncomfortable with it: should the
TupleTableSlot be part of the ModifyTableState rather than the EState?

> It also addresses the problem I mentioned previously that once
> tuple-routing is done, we failed to switch to a slot with the leaf
> partition's tupdesc (IOW, continued to use the original slot with root
> table's tupdesc causing spurious failures due to differences in attums
> between the leaf partition and the root table).
>
> Further patches 0002, 0003 and 0004 fix bugs that I sent one-big-patch for
> in my previous message.  Each patch has a test for the bug it's meant to fix.

Regarding 0002, I think that this is kind of a strange fix.  Wouldn't
it be better to get hold of the original tuple instead of reversing
the conversion?  And what of the idea of avoiding the conversion in
the (probably very common) case where we can?

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Sorry about the delay in replying.

On 2016/12/23 8:08, Robert Haas wrote:
> On Thu, Dec 22, 2016 at 3:35 AM, Amit Langote
> <Langote_Amit_f8@lab.ntt.co.jp> wrote:
>> While working on that, I discovered yet-another-bug having to do with the
>> tuple descriptor that's used as we route a tuple down a partition tree. If
>> attnums of given key attribute(s) are different on different levels, it
>> would be incorrect to use the original slot's (one passed by ExecInsert())
>> tuple descriptor to inspect the original slot's heap tuple, as we go down
>> the tree.  It might cause spurious "partition not found" at some level due
>> to looking at incorrect field in the input tuple because of using the
>> wrong tuple descriptor (root table's attnums not always same as other
>> partitioned tables in the tree).  Patch 0001 fixes that including a test.
> 
> I committed this, but I'm a bit uncomfortable with it: should the
> TupleTableSlot be part of the ModifyTableState rather than the EState?

Done that way in 0001 of the attached patches.  So, instead of making the
standalone partition_tuple_slot a field of EState (with the actual
TupleTableSlot in its tupleTable), it is now allocated within
ModifyTableState and CopyState, and released when ModifyTable node or
CopyFrom finishes, respectively.

>> It also addresses the problem I mentioned previously that once
>> tuple-routing is done, we failed to switch to a slot with the leaf
>> partition's tupdesc (IOW, continued to use the original slot with root
>> table's tupdesc causing spurious failures due to differences in attums
>> between the leaf partition and the root table).
>>
>> Further patches 0002, 0003 and 0004 fix bugs that I sent one-big-patch for
>> in my previous message.  Each patch has a test for the bug it's meant to fix.
> 
> Regarding 0002, I think that this is kind of a strange fix.  Wouldn't
> it be better to get hold of the original tuple instead of reversing
> the conversion?  And what of the idea of avoiding the conversion in
> the (probably very common) case where we can?

To get hold of the original tuple, how about adding an argument orig_slot
to ExecConstraints()?  I've implemented that approach in the new 0002.

Regarding the possibility of avoiding the conversion in very common cases,
I think that could be done considering the following:  If the mapping from
the attribute numbers of the parent table to that of a child table is an
identity map, we don't need to convert tuples.  Currently however,
convert_tuples_by_name() also requires tdtypeid of the input and output
TupleDescs to be equal.  The reason cited for that is that we may fail to
"inject the right OID into the tuple datum" if the types don't match.  In
case of partitioning, hasoid status must match between the parent and its
partitions at all times, so the aforementioned condition is satisfied
without requiring that tdtypeid are same.  And oid column (if present) is
always located at a given position in HeapTuple, so need not map that.

Based on the above argument, patch 0006  teaches convert_tuples_by_name()
to *optionally* not require tdtypeid of input and output tuple descriptors
to be equal.  It's implemented by introducing a new argument to
convert_tuples_by_name() named 'consider_typeid'.  We pass 'false' only
for the partitioning cases.


(Perhaps, the following should be its own new thread)

I noticed that ExecProcessReturning() doesn't work properly after tuple
routing (example shows how returning tableoid currently fails but I
mention some other issues below):

create table p (a int, b int) partition by range (a);
create table p1 partition of p for values from (1) to (10);
insert into p values (1) returning tableoid::regclass, *;
 tableoid | a | b
----------+---+---
 -        | 1 |
(1 row)

INSERT 0 1

I tried to fix that in 0007 to get:

insert into p values (1) returning tableoid::regclass, *;
 tableoid | a | b
----------+---+---
 p        | 1 |
(1 row)

INSERT 0 1

But I think it *may* be wrong to return the root table OID for tuples
inserted into leaf partitions, because with select we get partition OIDs:

select tableoid::regclass, * from p;
 tableoid | a | b
----------+---+---
 p1       | 1 |
(1 row)

If so, that means we should build the projection info (corresponding to
the returning list) for each target partition somehow.  ISTM, that's going
to have to be done within the planner by appropriate inheritance
translation of the original returning targetlist.

Thanks,
Amit

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 2016/12/26 19:46, Amit Langote wrote:
> (Perhaps, the following should be its own new thread)
> 
> I noticed that ExecProcessReturning() doesn't work properly after tuple
> routing (example shows how returning tableoid currently fails but I
> mention some other issues below):
> 
> create table p (a int, b int) partition by range (a);
> create table p1 partition of p for values from (1) to (10);
> insert into p values (1) returning tableoid::regclass, *;
>  tableoid | a | b
> ----------+---+---
>  -        | 1 |
> (1 row)
> 
> INSERT 0 1
> 
> I tried to fix that in 0007 to get:
> 
> insert into p values (1) returning tableoid::regclass, *;
>  tableoid | a | b
> ----------+---+---
>  p        | 1 |
> (1 row)
> 
> INSERT 0 1
> 
> But I think it *may* be wrong to return the root table OID for tuples
> inserted into leaf partitions, because with select we get partition OIDs:
> 
> select tableoid::regclass, * from p;
>  tableoid | a | b
> ----------+---+---
>  p1       | 1 |
> (1 row)
> 
> If so, that means we should build the projection info (corresponding to
> the returning list) for each target partition somehow.  ISTM, that's going
> to have to be done within the planner by appropriate inheritance
> translation of the original returning targetlist.

Turns out getting the 2nd result may not require planner tweaks after all.
Unless I'm missing something, translation of varattnos of the RETURNING
target list can be done as late as ExecInitModifyTable() for the insert
case, unlike update/delete (which do require planner's attention).

I updated the patch 0007 to implement the same, including the test. While
doing that, I realized map_partition_varattnos introduced in 0003 is
rather restrictive in its applicability, because it assumes varno = 1 for
the expressions it accepts as input for the mapping.  Mapping returning
(target) list required modifying map_partition_varattnos to accept
target_varno as additional argument.  That way, we can map arbitrary
expressions from the parent attributes numbers to partition attribute
numbers for expressions not limited to partition constraints.

Patches 0001 to 0006 unchanged.

Thanks,
Amit

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 2016/12/27 18:30, Rajkumar Raghuwanshi wrote:
> Hi Amit,
> 
> I have pulled latest sources from git and tried to create multi-level
> partition,  getting a server crash, below are steps to reproduce. please
> check if it is reproducible in your machine also.
> 

[ ... ]

> postgres=# INSERT INTO test_ml SELECT i, i, to_char(i, 'FM0000') FROM
> generate_series(0, 599, 2) i;
> server closed the connection unexpectedly
>     This probably means the server terminated abnormally
>     before or while processing the request.
> The connection to the server was lost. Attempting reset: Failed.

Thanks for the example.  Looks like there was an oversight in my patch
that got committed as 2ac3ef7a01 [1].

Attached patch should fix the same.

Thanks,
Amit

[1]
https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=2ac3ef7a01df859c62d0a02333b646d65eaec5ff

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 2016/12/27 18:48, 高增琦 wrote:
> Hi ,
> 
> I tried "COPY FROM"  in the git version. It inserts rows to wrong partition.
> 
> step to reproduce:
> create table t(a int, b int) partition by range(a);
> create table t_p1 partition of t for values from (1) to (100);
> create table t_p2 partition of t for values from (100) to (200);
> create table t_p3 partition of t for values from (200) to (300);
> insert into t values(1,1);
> insert into t values(101,101);
> insert into t values(201,201);
> copy (select * from t) to '/tmp/test2.txt';
> copy t from '/tmp/test2.txt';
> select * from t_p1;
> 
> result:
> postgres=# select * from t_p1;
>   a  |  b
> -----+-----
>    1 |   1
>    1 |   1
>  101 | 101
>  201 | 201
> (4 rows)
> 
> I think the argument "BulkInsertState" used in CopyFrom/heap_insert
> is related to this problem. Please check it.

You're quite right.  Attached should fix that.

Thanks,
Amit

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 2016/12/27 19:07, Amit Langote wrote:
> Attached should fix that.

Here are the last two patches with additional information like other
patches.  Forgot to do that yesterday.

Thanks,
Amit

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 2017/01/03 19:04, Rajkumar Raghuwanshi wrote:
> On Tue, Dec 27, 2016 at 3:24 PM, Amit Langote wrote:
>>
>> Attached patch should fix the same.
> 
> I have applied attached patch, server crash for range is fixed, but still
> getting crash for multi-level list partitioning insert.
> 
> postgres=# CREATE TABLE test_ml_l (a int, b int, c varchar) PARTITION BY
> LIST(c);

[ ... ]

> postgres=# INSERT INTO test_ml_l SELECT i, i, to_char(i/50, 'FM0000') FROM
> generate_series(0, 599, 2) i;
> server closed the connection unexpectedly
>     This probably means the server terminated abnormally
>     before or while processing the request.
> The connection to the server was lost. Attempting reset: Failed.

Hm, that's odd.  I tried your new example, but didn't get the crash.

Thanks,
Amit