Thread: [HACKERS] Default Partition for Range

[HACKERS] Default Partition for Range

From
Beena Emerson
Date:
Hello,

Many were in favour of the default partition for tables partitioned by range [1]. 
Please find attached the WIP patch for the same which can be applied over the default_partition_v12.patch.

Syntax: Same as agreed for list:
CREATE PARTITION <part_name> PARTITION OF <parent_tbl> DEFAULT;

Default Constraint:
Negation constraints of all existing partitions.

One case:
CREATE TABLE range1 (a int, b int) PARTITION by range (a);
CREATE TABLE range1_1 PARTITION OF range1 FOR VALUES FROM (1) TO (5);
CREATE TABLE range1_2 PARTITION OF range1 FOR VALUES FROM (7) TO (10);
CREATE TABLE range1_def PARTITION OF range1 DEFAULT;
\d+ range1_def
                                Table "public.range1_def"
 Column |  Type   | Collation | Nullable | Default | Storage | Stats target | Description
--------+---------+-----------+----------+---------+---------+--------------+-------------
 a      | integer |           | not null |         | plain   |              |
 b      | integer |           |          |         | plain   |              |
Partition of: range1 DEFAULT
Partition constraint: (((a < 1) OR (a >= 5)) AND ((a < 7) OR (a >= 10)))

It still needs more work: 
1. Handling addition of new partition after default, insertion of data, few more bugs
2. Documentation
3. Regression tests


--

Beena Emerson

EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
Attachment

Re: [HACKERS] Default Partition for Range

From
Ashutosh Bapat
Date:
On Mon, May 22, 2017 at 7:27 AM, Beena Emerson <memissemerson@gmail.com> wrote:
> Hello,
>
> Many were in favour of the default partition for tables partitioned by range
> [1].
> Please find attached the WIP patch for the same which can be applied over
> the default_partition_v12.patch.
>
> Syntax: Same as agreed for list:
> CREATE PARTITION <part_name> PARTITION OF <parent_tbl> DEFAULT;
>
> Default Constraint:
> Negation constraints of all existing partitions.
>
> One case:
> CREATE TABLE range1 (a int, b int) PARTITION by range (a);
> CREATE TABLE range1_1 PARTITION OF range1 FOR VALUES FROM (1) TO (5);
> CREATE TABLE range1_2 PARTITION OF range1 FOR VALUES FROM (7) TO (10);
> CREATE TABLE range1_def PARTITION OF range1 DEFAULT;
> \d+ range1_def
>                                 Table "public.range1_def"
>  Column |  Type   | Collation | Nullable | Default | Storage | Stats target
> | Description
> --------+---------+-----------+----------+---------+---------+--------------+-------------
>  a      | integer |           | not null |         | plain   |
> |
>  b      | integer |           |          |         | plain   |
> |
> Partition of: range1 DEFAULT
> Partition constraint: (((a < 1) OR (a >= 5)) AND ((a < 7) OR (a >= 10)))

Would it be more readable if this reads as NOT
(constraint_for_partition_1 || constraint_for_partition_2 ||
constraint_for_partition_3)? That way, the code to create
constraint_for_partition_n can be reused and there's high chance that
we will end up with consistent constraints?

>
> It still needs more work:
> 1. Handling addition of new partition after default, insertion of data, few
> more bugs
> 2. Documentation
> 3. Regression tests
>

I think, the default partition support for all the strategies
supporting it should be a single patch since most of the code will be
shared?
-- 
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company



Re: [HACKERS] Default Partition for Range

From
Beena Emerson
Date:
Hello,

On Mon, May 22, 2017 at 11:29 AM, Ashutosh Bapat <ashutosh.bapat@enterprisedb.com> wrote:
On Mon, May 22, 2017 at 7:27 AM, Beena Emerson <memissemerson@gmail.com> wrote:
Would it be more readable if this reads as NOT
(constraint_for_partition_1 || constraint_for_partition_2 ||
constraint_for_partition_3)? That way, the code to create
constraint_for_partition_n can be reused and there's high chance that
we will end up with consistent constraints?

PFA the patch which gives the default partition constraint as you have suggested.
 

>
> It still needs more work:
> 1. Handling addition of new partition after default, insertion of data, few
> more bugs
> 2. Documentation
> 3. Regression tests
>

I think, the default partition support for all the strategies
supporting it should be a single patch since most of the code will be
shared?


Dependency on list default patch:
gram.y : adding the syntax
partition.c:
- default_index member in PartitionBoundInfoData; 
- check_new_partition_bound : the code for adding a partition after default has been completely reused.
- isDefaultPartitionBound function is used.

The structures are same  but the handling of list and range is very different and the code mostly has  the switch case construct to handle the 2 separately. I feel it could be taken separately.

As suggested in the default list thread, I have created a partition_bound_has_default macro and avoided usage of has_default in range code. This has to be used for list as well.
Another suggestion for list which has to be implemented in this patch in removal of PARTITION_DEFAULT. Ii have not done this in this version.

--

Beena Emerson

EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
Attachment

Re: [HACKERS] Default Partition for Range

From
Jeevan Ladhe
Date:
Hi Beena,

I went through your patch, and here are some of my comments:

- For generating a qual for default range partition, instead of scanning for all
the children and collecting all the boundspecs, how about creating and negating
an expression from the lists of lowerdatums and upperdatums in boundinfo.   

- Wrong comment:
+ int default_index; /* Index of the default list partition. */

- Suggested by Robert earlier on default list partitioning thread, instead of
abbreviating is_def/found_def use is(found)_default etc.

- unrelated change:
- List           *all_parts;
+ List   *all_parts;

- typo: partiton should be partition, similar typo at other places too.
+  * A non-default range partiton table does not currently allow partition keys

- Useless hunk for this patch:
- Oid        defid;
+ Oid defid;

- better to use IsA here, instead of doing explicit check:
+ bound->content[i] = (datum->type == T_DefElem)
+ ? RANGE_DATUM_DEFAULT

- It is better to use head of either lowerdatums or upperdatums list to verify
if its a default partition and get rid of the constant PARTITION_DEFAULT
altogether.

+ {
+ /*
+ * If the partition is the default partition switch back to
+ * PARTITION_STRATEGY_RANGE
+ */
+ if (spec->strategy == PARTITION_DEFAULT)
+ {
+ is_def = true;
+ result_spec->strategy = PARTITION_STRATEGY_RANGE;
+ }

- I am sorry, but I could not understand following hunk. Does this change really
belongs to this patch? If not, it will be better to handle it separately.

@@ -2242,33 +2387,16 @@ get_partition_for_tuple(PartitionDispatch *pd,
  ecxt->ecxt_scantuple = slot;
  FormPartitionKeyDatum(parent, slot, estate, values, isnull);
 
- if (key->strategy == PARTITION_STRATEGY_RANGE)
+ if (key->strategy == PARTITION_STRATEGY_LIST && isnull[0])
  {
  /*
- * Since we cannot route tuples with NULL partition keys through
- * a range-partitioned table, simply return that no partition
- * exists
+ * A null partition key is only acceptable if null-accepting list
+ * partition exists.
  */
- for (i = 0; i < key->partnatts; i++)
- {
- if (isnull[i])
- {
- *failed_at = parent;
- *failed_slot = slot;
- result = -1;
- goto error_exit;
- }
- }
+ if (partition_bound_accepts_nulls(partdesc->boundinfo))
+ cur_index = partdesc->boundinfo->null_index;

- Change not related to this patch:
- List           *all_parts;
+ List   *all_parts;

- In the function get_qual_from_partbound() is_def is always going to be false
for range partition, the function get_qual_for_range can be directly passed
false instead.

- Following comment for function get_qual_for_range_default() implies that this
function returns bool, but the actually it returns a List.
+ *
+ * If DEFAULT is the only partiton for the table then this returns TRUE.
+ *


Regards,
Jeevan Ladhe

On Wed, May 24, 2017 at 12:52 AM, Beena Emerson <memissemerson@gmail.com> wrote:
Hello,

On Mon, May 22, 2017 at 11:29 AM, Ashutosh Bapat <ashutosh.bapat@enterprisedb.com> wrote:
On Mon, May 22, 2017 at 7:27 AM, Beena Emerson <memissemerson@gmail.com> wrote:
Would it be more readable if this reads as NOT
(constraint_for_partition_1 || constraint_for_partition_2 ||
constraint_for_partition_3)? That way, the code to create
constraint_for_partition_n can be reused and there's high chance that
we will end up with consistent constraints?

PFA the patch which gives the default partition constraint as you have suggested.
 

>
> It still needs more work:
> 1. Handling addition of new partition after default, insertion of data, few
> more bugs
> 2. Documentation
> 3. Regression tests
>

I think, the default partition support for all the strategies
supporting it should be a single patch since most of the code will be
shared?


Dependency on list default patch:
gram.y : adding the syntax
partition.c:
- default_index member in PartitionBoundInfoData; 
- check_new_partition_bound : the code for adding a partition after default has been completely reused.
- isDefaultPartitionBound function is used.

The structures are same  but the handling of list and range is very different and the code mostly has  the switch case construct to handle the 2 separately. I feel it could be taken separately.

As suggested in the default list thread, I have created a partition_bound_has_default macro and avoided usage of has_default in range code. This has to be used for list as well.
Another suggestion for list which has to be implemented in this patch in removal of PARTITION_DEFAULT. Ii have not done this in this version.

--

Beena Emerson

EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Default Partition for Range

From
Beena Emerson
Date:
On Mon, May 29, 2017 at 3:22 PM, Jeevan Ladhe
<jeevan.ladhe@enterprisedb.com> wrote:
> Hi Beena,
>
> I went through your patch, and here are some of my comments:
>

Thank you for your comments. I  will take care of them in the next
version of patch.

> - I am sorry, but I could not understand following hunk. Does this change
> really
> belongs to this patch? If not, it will be better to handle it separately.
>
> @@ -2242,33 +2387,16 @@ get_partition_for_tuple(PartitionDispatch *pd,
>   ecxt->ecxt_scantuple = slot;
>   FormPartitionKeyDatum(parent, slot, estate, values, isnull);
>
> - if (key->strategy == PARTITION_STRATEGY_RANGE)
> + if (key->strategy == PARTITION_STRATEGY_LIST && isnull[0])
>   {
>   /*
> - * Since we cannot route tuples with NULL partition keys through
> - * a range-partitioned table, simply return that no partition
> - * exists
> + * A null partition key is only acceptable if null-accepting list
> + * partition exists.
>   */

In RANGE, initially NULL was not allowed, now  NULL is routed to
default. I have only removed the check for null in RANGE and kept the
check for null partition in case of list.

-- 

Beena Emerson

EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company



Re: [HACKERS] Default Partition for Range

From
Beena Emerson
Date:
Hello,

PFA the updated patch.
Dependent patch default_partition_v17.patch [1]
This patch adds regression tests and removes the separate function to
get default partition qual.


On Mon, May 29, 2017 at 3:22 PM, Jeevan Ladhe
<jeevan.ladhe@enterprisedb.com> wrote:
> Hi Beena,
>
> I went through your patch, and here are some of my comments:
>
> - For generating a qual for default range partition, instead of scanning for
> all
> the children and collecting all the boundspecs, how about creating and
> negating
> an expression from the lists of lowerdatums and upperdatums in boundinfo.
>

Unlike list, range partition can be for multiple columns and the
expressions get complicated. I have used the same logic of looping
through different partitions and negating the ORed expr. However, this
is now done under get_qual_for_range.


> - Wrong comment:
> + int default_index; /* Index of the default list partition. */

Comment is made generic in the dependent patch.

>
> - Suggested by Robert earlier on default list partitioning thread, instead
> of
> abbreviating is_def/found_def use is(found)_default etc.

corrected.

>
> - unrelated change:
> - List           *all_parts;
> + List   *all_parts;
>

undone.

> - typo: partiton should be partition, similar typo at other places too.
> +  * A non-default range partiton table does not currently allow partition
> keys
>

rectified.

> - Useless hunk for this patch:
> - Oid        defid;
> + Oid defid;

undone.

>
> - better to use IsA here, instead of doing explicit check:
> + bound->content[i] = (datum->type == T_DefElem)
> + ? RANGE_DATUM_DEFAULT

Modified

>
> - It is better to use head of either lowerdatums or upperdatums list to
> verify
> if its a default partition and get rid of the constant PARTITION_DEFAULT
> altogether.
>

modified this part as necessary.


> - In the function get_qual_from_partbound() is_def is always going to be
> false
> for range partition, the function get_qual_for_range can be directly passed
> false instead.
>
> - Following comment for function get_qual_for_range_default() implies that
> this
> function returns bool, but the actually it returns a List.
> + *
> + * If DEFAULT is the only partiton for the table then this returns TRUE.
> + *
>
Updated.

[1] http://www.mail-archive.com/pgsql-hackers@postgresql.org/msg315573.html

-- 

Beena Emerson

EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Attachment

Re: [HACKERS] Default Partition for Range

From
Rafia Sabih
Date:
On Wed, May 31, 2017 at 5:53 PM, Beena Emerson <memissemerson@gmail.com> wrote:
> Hello,
>
> PFA the updated patch.
> Dependent patch default_partition_v17.patch [1]
> This patch adds regression tests and removes the separate function to
> get default partition qual.
>
>
> On Mon, May 29, 2017 at 3:22 PM, Jeevan Ladhe
> <jeevan.ladhe@enterprisedb.com> wrote:
>> Hi Beena,
>>
>> I went through your patch, and here are some of my comments:
>>
>> - For generating a qual for default range partition, instead of scanning for
>> all
>> the children and collecting all the boundspecs, how about creating and
>> negating
>> an expression from the lists of lowerdatums and upperdatums in boundinfo.
>>
>
> Unlike list, range partition can be for multiple columns and the
> expressions get complicated. I have used the same logic of looping
> through different partitions and negating the ORed expr. However, this
> is now done under get_qual_for_range.
>
>
>> - Wrong comment:
>> + int default_index; /* Index of the default list partition. */
>
> Comment is made generic in the dependent patch.
>
>>
>> - Suggested by Robert earlier on default list partitioning thread, instead
>> of
>> abbreviating is_def/found_def use is(found)_default etc.
>
> corrected.
>
>>
>> - unrelated change:
>> - List           *all_parts;
>> + List   *all_parts;
>>
>
> undone.
>
>> - typo: partiton should be partition, similar typo at other places too.
>> +  * A non-default range partiton table does not currently allow partition
>> keys
>>
>
> rectified.
>
>> - Useless hunk for this patch:
>> - Oid        defid;
>> + Oid defid;
>
> undone.
>
>>
>> - better to use IsA here, instead of doing explicit check:
>> + bound->content[i] = (datum->type == T_DefElem)
>> + ? RANGE_DATUM_DEFAULT
>
> Modified
>
>>
>> - It is better to use head of either lowerdatums or upperdatums list to
>> verify
>> if its a default partition and get rid of the constant PARTITION_DEFAULT
>> altogether.
>>
>
> modified this part as necessary.
>
>
>> - In the function get_qual_from_partbound() is_def is always going to be
>> false
>> for range partition, the function get_qual_for_range can be directly passed
>> false instead.
>>
>> - Following comment for function get_qual_for_range_default() implies that
>> this
>> function returns bool, but the actually it returns a List.
>> + *
>> + * If DEFAULT is the only partiton for the table then this returns TRUE.
>> + *
>>
> Updated.
>
> [1] http://www.mail-archive.com/pgsql-hackers@postgresql.org/msg315573.html
>

Hi Beena,

I had a look at the patch from the angle of aesthetics and there are a
few cosmetic changes I might suggest. Please have a look at the
attached patch and if you agree with those changes you may merge it on
your patch. The patch also fixes a couple of more spelling mistakes
unrelated to this patch.

-- 
Regards,
Rafia Sabih
EnterpriseDB: http://www.enterprisedb.com/

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Attachment

Re: [HACKERS] Default Partition for Range

From
Amit Kapila
Date:
On Fri, Jun 2, 2017 at 4:12 PM, Rafia Sabih
<rafia.sabih@enterprisedb.com> wrote:
> On Wed, May 31, 2017 at 5:53 PM, Beena Emerson <memissemerson@gmail.com> wrote:
>
> Hi Beena,
>
> I had a look at the patch from the angle of aesthetics and there are a
> few cosmetic changes I might suggest. Please have a look at the
> attached patch and if you agree with those changes you may merge it on
> your patch. The patch also fixes a couple of more spelling mistakes
> unrelated to this patch.
>

I think if you have found spelling mistakes unrelated to this patch,
then it is better to submit those as a separate patch in a new thread.


-- 
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com



Re: [HACKERS] Default Partition for Range

From
Robert Haas
Date:
On Fri, Jun 2, 2017 at 8:09 AM, Amit Kapila <amit.kapila16@gmail.com> wrote:
> I think if you have found spelling mistakes unrelated to this patch,
> then it is better to submit those as a separate patch in a new thread.

+1.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company



Re: [HACKERS] Default Partition for Range

From
Rafia Sabih
Date:
On Fri, Jun 2, 2017 at 5:48 PM, Robert Haas <robertmhaas@gmail.com> wrote:
> On Fri, Jun 2, 2017 at 8:09 AM, Amit Kapila <amit.kapila16@gmail.com> wrote:
>> I think if you have found spelling mistakes unrelated to this patch,
>> then it is better to submit those as a separate patch in a new thread.
>
> +1.

Sure, attached is the version without those changes.

-- 
Regards,
Rafia Sabih
EnterpriseDB: http://www.enterprisedb.com/

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Attachment

Re: [HACKERS] Default Partition for Range

From
Beena Emerson
Date:
Hello,

On Sun, Jun 4, 2017 at 9:26 AM, Rafia Sabih
<rafia.sabih@enterprisedb.com> wrote:
> On Fri, Jun 2, 2017 at 5:48 PM, Robert Haas <robertmhaas@gmail.com> wrote:
>> On Fri, Jun 2, 2017 at 8:09 AM, Amit Kapila <amit.kapila16@gmail.com> wrote:
>>> I think if you have found spelling mistakes unrelated to this patch,
>>> then it is better to submit those as a separate patch in a new thread.
>>
>> +1.
>
> Sure, attached is the version without those changes.
>

Thank you for your comments. I have applied most of the changes. The
regression comment change from 'fail' -> ' Following statement should
fail' and 'ok' -> 'Following should complete successfully' is ignored
since other tests in the file had similar comments

The new patch is rebased over default_partition_v18.patch
[http://www.mail-archive.com/pgsql-hackers@postgresql.org/msg315831.html]


-- 

Beena Emerson

EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Attachment

Re: [HACKERS] Default Partition for Range

From
Dilip Kumar
Date:
On Mon, Jun 5, 2017 at 1:43 AM, Beena Emerson <memissemerson@gmail.com> wrote:
> The new patch is rebased over default_partition_v18.patch
> [http://www.mail-archive.com/pgsql-hackers@postgresql.org/msg315831.html]

I have done the initial review of the patch, I have few comments.


+ if ((lower->content[0] == RANGE_DATUM_DEFAULT))
+ {
+ if (partition_bound_has_default(partdesc->boundinfo))
+ {
+ overlap = true;
+ with = partdesc->boundinfo->default_index;
+ }

I think we can change if ((lower->content[0] == RANGE_DATUM_DEFAULT))
check to if (spec->is_default) that way it will be consistent with the
check in the PARTITION_STRATEGY_LIST.  Or we can move this complete
check outside of the switch.

-------------

- PartitionRangeDatum *datum = castNode(PartitionRangeDatum, lfirst(lc));
+ Node   *node = lfirst(lc);
+ PartitionRangeDatum *datum;
+
+ datum = castNode(PartitionRangeDatum, node);

Why do you need to change this?

--------------

+ if (!datums)
+ bound->content[i] = RANGE_DATUM_DEFAULT;
+

Better to check if (datums != NULL) for non boolean types.

-------------
+ if (content1[i] == RANGE_DATUM_DEFAULT ||
+ content2[i] == RANGE_DATUM_DEFAULT)
+ {
+ if (content1[i] == content2[i])
+ return 0;
+ else if (content1[i] == RANGE_DATUM_DEFAULT)
+ return -1;

I don't see any comments why default partition should be considered
smallest in the index comparison. For negative infinity, it's pretty
obvious by the enum name itself.

-------------

-- 
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com



Re: [HACKERS] Default Partition for Range

From
Beena Emerson
Date:
Hello Dilip,

On Mon, Jun 5, 2017 at 8:44 PM, Dilip Kumar <dilipbalaut@gmail.com> wrote:
> On Mon, Jun 5, 2017 at 1:43 AM, Beena Emerson <memissemerson@gmail.com> wrote:
>> The new patch is rebased over default_partition_v18.patch
>> [http://www.mail-archive.com/pgsql-hackers@postgresql.org/msg315831.html]
>
> I have done the initial review of the patch, I have few comments.
>
Thank you.
>
> + if ((lower->content[0] == RANGE_DATUM_DEFAULT))
> + {
> + if (partition_bound_has_default(partdesc->boundinfo))
> + {
> + overlap = true;
> + with = partdesc->boundinfo->default_index;
> + }
>
> I think we can change if ((lower->content[0] == RANGE_DATUM_DEFAULT))
> check to if (spec->is_default) that way it will be consistent with the
> check in the PARTITION_STRATEGY_LIST.  Or we can move this complete
> check outside of the switch.

I have now moved the is_default check for both list and range outside
the switch case.

>
> -------------
>
> - PartitionRangeDatum *datum = castNode(PartitionRangeDatum, lfirst(lc));
> + Node   *node = lfirst(lc);
> + PartitionRangeDatum *datum;
> +
> + datum = castNode(PartitionRangeDatum, node);
>
> Why do you need to change this?

I forgot to remove it.
It was needed for previous version of patch but no longer needed and
hence reverted this change.

>
> --------------
>
> + if (!datums)
> + bound->content[i] = RANGE_DATUM_DEFAULT;
> +
>
> Better to check if (datums != NULL) for non boolean types.

done.

>
> -------------
> + if (content1[i] == RANGE_DATUM_DEFAULT ||
> + content2[i] == RANGE_DATUM_DEFAULT)
> + {
> + if (content1[i] == content2[i])
> + return 0;
> + else if (content1[i] == RANGE_DATUM_DEFAULT)
> + return -1;
>
> I don't see any comments why default partition should be considered
> smallest in the index comparison. For negative infinity, it's pretty
> obvious by the enum name itself.

Default could be lowest or highest, no specific reason for putting it lowest.
I have not added any comments in this version.

--

Beena Emerson

EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Attachment

Re: [HACKERS] Default Partition for Range

From
Beena Emerson
Date:
Hello,

PFA the updated patch.
This is rebased over v21 patches of list partition.
(http://www.mail-archive.com/pgsql-hackers@postgresql.org/msg316818.html)


-- 

Beena Emerson

EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Attachment

Re: [HACKERS] Default Partition for Range

From
Dilip Kumar
Date:
On Thu, Jun 15, 2017 at 11:20 AM, Beena Emerson <memissemerson@gmail.com> wrote:
> Hello,
>
> PFA the updated patch.
> This is rebased over v21 patches of list partition.
> (http://www.mail-archive.com/pgsql-hackers@postgresql.org/msg316818.html)

While testing I have noticed segmentation fault for a simple case.

create table test(a int, b int) partition by range(a,b);
create table test1 partition of test DEFAULT;
postgres=# drop table test1;
server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.

This is basically crashing in RelationBuildPartitionDesc, so I think
we don't have any test case for testing DEFAULT range partition where
partition key has more than one attribute.  So I suggest we can add
such test case.


-- 
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com



Re: [HACKERS] Default Partition for Range

From
Dilip Kumar
Date:
On Tue, Jun 20, 2017 at 6:57 PM, Dilip Kumar <dilipbalaut@gmail.com> wrote:
> This is basically crashing in RelationBuildPartitionDesc, so I think
> we don't have any test case for testing DEFAULT range partition where
> partition key has more than one attribute.  So I suggest we can add
> such test case.

Some more comments.

<code>
- bound->datums = (Datum *) palloc0(key->partnatts * sizeof(Datum));
- bound->content = (RangeDatumContent *) palloc0(key->partnatts *
-   sizeof(RangeDatumContent));
+ bound->datums = datums ? (Datum *) palloc0(key->partnatts * sizeof(Datum))
+ : NULL;
+ bound->content = (RangeDatumContent *) palloc0(
+ (datums ? key->partnatts : 1) * sizeof(RangeDatumContent)); bound->lower = lower;
 i = 0;
+
+ /* If default, datums are NULL */
+ if (datums == NULL)
+ bound->content[i] = RANGE_DATUM_DEFAULT;
</code>

For the default partition we are only setting bound->content[0] to
default, but content for others key
attributes are not initialized.  But later in the code, if the content
of the first key is RANGE_DATUM_DEFAULT then it should not access the
next content,  but I see there are some exceptions.  Which can access
uninitialized value?

for example see below code

<code>
for (i = 0; i < key->partnatts; i++) {
+ if (rb_content[i] == RANGE_DATUM_DEFAULT)   --> why it's going to
content for next parttion attribute, we never initialized that?
+ continue;
+ if (rb_content[i] != RANGE_DATUM_FINITE) return rb_content[i] == RANGE_DATUM_NEG_INF ? -1 : 1;
</code>

Also
In RelatiobBuildPartitionDesc

<code>
for (j = 0; j < key->partnatts; j++)
{
-- Suppose first is RANGE_DATUM_DEFAULT then we should not check next  because that is never initialized.  I think this
isthe cause of
 
the crash also what I have reported above.
----
if (rbounds[i]->content[j] == RANGE_DATUM_FINITE)
boundinfo->datums[i][j] =
datumCopy(rbounds[i]->datums[j],key->parttypbyval[j],key->parttyplen[j]);
/* Remember, we are storing the tri-state value. */
boundinfo->content[i][j] = rbounds[i]->content[j];
</code>

-- 
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com



Re: [HACKERS] Default Partition for Range

From
Robert Haas
Date:
On Wed, Jun 21, 2017 at 8:57 AM, Dilip Kumar <dilipbalaut@gmail.com> wrote:
> For the default partition we are only setting bound->content[0] to
> default, but content for others key
> attributes are not initialized.  But later in the code, if the content
> of the first key is RANGE_DATUM_DEFAULT then it should not access the
> next content,  but I see there are some exceptions.  Which can access
> uninitialized value?

I think somebody should do some testing of the existing code with
valgrind.  And then apply the list-partitioning patch and this patch,
and do some more testing with valgrind.  It seems to be really easy to
miss these uninitialized access problems during code review.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company



Re: [HACKERS] Default Partition for Range

From
Dilip Kumar
Date:
On Wed, Jun 21, 2017 at 8:08 PM, Robert Haas <robertmhaas@gmail.com> wrote:
> I think somebody should do some testing of the existing code with
> valgrind.  And then apply the list-partitioning patch and this patch,
> and do some more testing with valgrind.  It seems to be really easy to
> miss these uninitialized access problems during code review.

I think it will help,  but it may not help in all the scenarios
because most of the places we are allocating memory with palloc0 (
initialized with 0) but it never initialized with RANGE_DATUM_DEFAULT
except the first element (in the case of DEFAULT partition).  And,
later they may be considered as RANGE_DATUM_FINITE (because its value
is 0).

One solution can be that if bound is DEFAULT then initialize with
RANGE_DATUM_DEFAULT for the complete content array for that partition
bound instead of just first.  Otherwise, we need to be careful of
early exiting wherever we are looping the content array of the DEFAULT
bound.

-- 
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com



Re: [HACKERS] Default Partition for Range

From
Rahila Syed
Date:
Hi Beena,

I started testing and reviewing the patch. Can you update the patch as v5 patch does not apply cleanly on master?

Thank you,
Rahila Syed

On Wed, Jun 21, 2017 at 8:43 PM, Dilip Kumar <dilipbalaut@gmail.com> wrote:
On Wed, Jun 21, 2017 at 8:08 PM, Robert Haas <robertmhaas@gmail.com> wrote:
> I think somebody should do some testing of the existing code with
> valgrind.  And then apply the list-partitioning patch and this patch,
> and do some more testing with valgrind.  It seems to be really easy to
> miss these uninitialized access problems during code review.

I think it will help,  but it may not help in all the scenarios
because most of the places we are allocating memory with palloc0 (
initialized with 0) but it never initialized with RANGE_DATUM_DEFAULT
except the first element (in the case of DEFAULT partition).  And,
later they may be considered as RANGE_DATUM_FINITE (because its value
is 0).

One solution can be that if bound is DEFAULT then initialize with
RANGE_DATUM_DEFAULT for the complete content array for that partition
bound instead of just first.  Otherwise, we need to be careful of
early exiting wherever we are looping the content array of the DEFAULT
bound.

--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Default Partition for Range

From
Beena Emerson
Date:
On Wed, Jun 28, 2017 at 12:34 PM, Rahila Syed <rahilasyed90@gmail.com> wrote:
> Hi Beena,
>
> I started testing and reviewing the patch. Can you update the patch as v5
> patch does not apply cleanly on master?
>

I am currently working on Dilip's comments, I will update the patch soon.

-- 

Beena Emerson

EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company



Re: [HACKERS] Default Partition for Range

From
Beena Emerson
Date:
Hello Rahila,

On Wed, Jun 28, 2017 at 12:34 PM, Rahila Syed <rahilasyed90@gmail.com> wrote:
> Hi Beena,
>
> I started testing and reviewing the patch. Can you update the patch as v5
> patch does not apply cleanly on master?
>

Thanks for looking into this.

The patch is to be applied on Jeevn's partition patch
(http://www.mail-archive.com/pgsql-hackers@postgresql.org/msg316818.html)
which can be applied over commit
a12c09ad86e60a8acb269744b8ee86429dda2cd8.


-- 

Beena Emerson

EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company



Re: [HACKERS] Default Partition for Range

From
Beena Emerson
Date:
Hello Dilip,

On Wed, Jun 21, 2017 at 6:27 PM, Dilip Kumar <dilipbalaut@gmail.com> wrote:
> On Tue, Jun 20, 2017 at 6:57 PM, Dilip Kumar <dilipbalaut@gmail.com> wrote:
>> This is basically crashing in RelationBuildPartitionDesc, so I think
>> we don't have any test case for testing DEFAULT range partition where
>> partition key has more than one attribute.  So I suggest we can add
>> such test case.
>
> Some more comments.
>
> <code>
> - bound->datums = (Datum *) palloc0(key->partnatts * sizeof(Datum));
> - bound->content = (RangeDatumContent *) palloc0(key->partnatts *
> -   sizeof(RangeDatumContent));
> + bound->datums = datums ? (Datum *) palloc0(key->partnatts * sizeof(Datum))
> + : NULL;
> + bound->content = (RangeDatumContent *) palloc0(
> + (datums ? key->partnatts : 1) * sizeof(RangeDatumContent));
>   bound->lower = lower;
>
>   i = 0;
> +
> + /* If default, datums are NULL */
> + if (datums == NULL)
> + bound->content[i] = RANGE_DATUM_DEFAULT;
> </code>
>
> For the default partition we are only setting bound->content[0] to
> default, but content for others key
> attributes are not initialized.  But later in the code, if the content
> of the first key is RANGE_DATUM_DEFAULT then it should not access the
> next content,  but I see there are some exceptions.  Which can access
> uninitialized value?
>
> for example see below code
>
> <code>
> for (i = 0; i < key->partnatts; i++)
>   {
> + if (rb_content[i] == RANGE_DATUM_DEFAULT)   --> why it's going to
> content for next parttion attribute, we never initialized that?
> + continue;
> +
>   if (rb_content[i] != RANGE_DATUM_FINITE)
>   return rb_content[i] == RANGE_DATUM_NEG_INF ? -1 : 1;
> </code>
>
> Also
> In RelatiobBuildPartitionDesc
>
> <code>
> for (j = 0; j < key->partnatts; j++)
> {
> -- Suppose first is RANGE_DATUM_DEFAULT then we should not check next
>    because that is never initialized.  I think this is the cause of
> the crash also what I have reported above.
> ----
> if (rbounds[i]->content[j] == RANGE_DATUM_FINITE)
> boundinfo->datums[i][j] =
> datumCopy(rbounds[i]->datums[j],
>  key->parttypbyval[j],
>  key->parttyplen[j]);
> /* Remember, we are storing the tri-state value. */
> boundinfo->content[i][j] = rbounds[i]->content[j];
> </code>
>

Thank you for your review and analysis.

I have updated the patch.
- bound->content is set to RANGE_DATUM_DEFAULT for each of the keys
and not just the first one.
- Improve the way of handling DEFAULT bounds in RelationBuildPartitionDesc,

There is a test for multiple column range in alter_table.sql

-- 

Beena Emerson

EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Attachment

Re: [HACKERS] Default Partition for Range

From
Dilip Kumar
Date:
On Fri, Jun 30, 2017 at 11:56 AM, Beena Emerson <memissemerson@gmail.com> wrote:
> Hello Dilip,
>
> On Wed, Jun 21, 2017 at 6:27 PM, Dilip Kumar <dilipbalaut@gmail.com> wrote:
>> On Tue, Jun 20, 2017 at 6:57 PM, Dilip Kumar <dilipbalaut@gmail.com> wrote:
>>> This is basically crashing in RelationBuildPartitionDesc, so I think

>
> Thank you for your review and analysis.
>
> I have updated the patch.
> - bound->content is set to RANGE_DATUM_DEFAULT for each of the keys
> and not just the first one.
> - Improve the way of handling DEFAULT bounds in RelationBuildPartitionDesc,
>
> There is a test for multiple column range in alter_table.sql

Thanks for update patch,  After this fix segmentation fault is solved.

I have some more comments.

1.

lower = make_one_range_bound(key, -1, spec->lowerdatums, true); upper = make_one_range_bound(key, -1,
spec->upperdatums,false);
 

+ /* Default case is not handled here */
+ if (spec->is_default)
+ break;
+

I think we can move default check just above the make range bound it
will avoid unnecessary processing.


2.
RelationBuildPartitionDesc
{
....
      /*
* If either of them has infinite element, we can't equate
* them.  Even when both are infinite, they'd have
* opposite signs, because only one of cur and prev is a
* lower bound).
*/
if (cur->content[j] != RANGE_DATUM_FINITE || prev->content[j] != RANGE_DATUM_FINITE)
{
is_distinct = true;
break;
}
After default range partitioning patch the above comment doesn't sound
very accurate and
can lead to the confusion, now here we are not only considering
infinite bounds but
there is also a possibility that prev bound can be DEFAULT, so
comments should clearly
mention that.

3.

+ bound->datums = datums ? (Datum *) palloc0(key->partnatts * sizeof(Datum))
+ : NULL; bound->content = (RangeDatumContent *) palloc0(key->partnatts *   sizeof(RangeDatumContent)); bound->lower =
lower;
 i = 0;
+
+ /* datums are NULL for default */
+ if (datums == NULL)
+ for (i = 0; i < key->partnatts; i++)
+ bound->content[i] = RANGE_DATUM_DEFAULT;

To me, it will look cleaner if we keep bound->content=NULL for
default, instead of allocating memory
and initializing each element,  But it's just a suggestions and you
can decide whichever way
seems better to you.  Then the other places e.g.
+ if (cur->content[i] == RANGE_DATUM_DEFAULT)
+ {
+ /*

will change like

+ if (cur->content == NULL)
+ {
+ /*

-- 
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com



Re: [HACKERS] Default Partition for Range

From
Rahila Syed
Date:

Hello Beena,

Thanks for the updated patch. It passes the basic tests which I performed.

Few comments:
1. In   check_new_partition_bound(),           

> /* Default case is not handled here */
>                if (spec->is_default)
>                    break;
The default partition check here can be performed in common for both range and list partition.

2.     RANGE_DATUM_FINITE = 0,     /* actual datum stored elsewhere */
+   RANGE_DATUM_DEFAULT,        /* default */

More elaborative comment?

3.  Inside make_one_range_bound(),

>+
>+   /* datums are NULL for default */
>+   if (datums == NULL)
>+       for (i = 0; i < key->partnatts; i++)
>+           bound->content[i] = RANGE_DATUM_DEFAULT;
>+
IMO, returning bound at this stage will make code clearer as further processing
does not happen for default partition.

4.
@@ -549,6 +568,7 @@ RelationBuildPartitionDesc(Relation rel)
                                                sizeof(RangeDatumContent *));
                    boundinfo->indexes = (int *) palloc((ndatums + 1) *
                                                        sizeof(int));
+                   boundinfo->default_index = -1;
This should also be done commonly for both default list and range partition.

5.
              if (!spec->is_default && partdesc->nparts > 0)
                {
                    ListCell   *cell;

                    Assert(boundinfo &&
                           boundinfo->strategy == PARTITION_STRATEGY_LIST &&
                           (boundinfo->ndatums > 0 ||
                            partition_bound_accepts_nulls(boundinfo) ||
                            partition_bound_has_default(boundinfo)));
The assertion condition partition_bound_has_default(boundinfo) is rendered useless
because of if condition change and can be removed perhaps. 

6. I think its better to have following  regression tests coverage

--Testing with inserting one NULL and other non NULL values for multicolumn range partitioned table.

postgres=# CREATE TABLE range_parted (
postgres(#     a int,
postgres(#     b int
postgres(# ) PARTITION BY RANGE (a, b);
CREATE TABLE
postgres=#
postgres=# CREATE TABLE part1 (
postgres(#     a int NOT NULL CHECK (a = 1),
postgres(#     b int NOT NULL CHECK (b >= 1 AND b <= 10)
postgres(# );
CREATE TABLE

postgres=# ALTER TABLE range_parted ATTACH PARTITION part1 FOR VALUES FROM (1, 1) TO (1, 10);
ALTER TABLE
postgres=# CREATE TABLE partr_def1 PARTITION OF range_parted DEFAULT;
CREATE TABLE

postgres=# insert into range_parted values (1,NULL);
INSERT 0 1
postgres=# insert into range_parted values (NULL,9);
INSERT 0 1

--Testing the boundary conditions like in the above example
 following should pass.

postgres=# insert into partr_def1 values (1,10);
INSERT 0 1

Thank you,
Rahila Syed










On Mon, Jul 3, 2017 at 8:00 PM, Dilip Kumar <dilipbalaut@gmail.com> wrote:
On Fri, Jun 30, 2017 at 11:56 AM, Beena Emerson <memissemerson@gmail.com> wrote:
> Hello Dilip,
>
> On Wed, Jun 21, 2017 at 6:27 PM, Dilip Kumar <dilipbalaut@gmail.com> wrote:
>> On Tue, Jun 20, 2017 at 6:57 PM, Dilip Kumar <dilipbalaut@gmail.com> wrote:
>>> This is basically crashing in RelationBuildPartitionDesc, so I think

>
> Thank you for your review and analysis.
>
> I have updated the patch.
> - bound->content is set to RANGE_DATUM_DEFAULT for each of the keys
> and not just the first one.
> - Improve the way of handling DEFAULT bounds in RelationBuildPartitionDesc,
>
> There is a test for multiple column range in alter_table.sql

Thanks for update patch,  After this fix segmentation fault is solved.

I have some more comments.

1.

lower = make_one_range_bound(key, -1, spec->lowerdatums, true);
  upper = make_one_range_bound(key, -1, spec->upperdatums, false);

+ /* Default case is not handled here */
+ if (spec->is_default)
+ break;
+

I think we can move default check just above the make range bound it
will avoid unnecessary processing.


2.
RelationBuildPartitionDesc
{
....

       /*
* If either of them has infinite element, we can't equate
* them.  Even when both are infinite, they'd have
* opposite signs, because only one of cur and prev is a
* lower bound).
*/
if (cur->content[j] != RANGE_DATUM_FINITE ||
  prev->content[j] != RANGE_DATUM_FINITE)
{
is_distinct = true;
break;
}
After default range partitioning patch the above comment doesn't sound
very accurate and
can lead to the confusion, now here we are not only considering
infinite bounds but
there is also a possibility that prev bound can be DEFAULT, so
comments should clearly
mention that.

3.

+ bound->datums = datums ? (Datum *) palloc0(key->partnatts * sizeof(Datum))
+ : NULL;
  bound->content = (RangeDatumContent *) palloc0(key->partnatts *
    sizeof(RangeDatumContent));
  bound->lower = lower;

  i = 0;
+
+ /* datums are NULL for default */
+ if (datums == NULL)
+ for (i = 0; i < key->partnatts; i++)
+ bound->content[i] = RANGE_DATUM_DEFAULT;

To me, it will look cleaner if we keep bound->content=NULL for
default, instead of allocating memory
and initializing each element,  But it's just a suggestions and you
can decide whichever way
seems better to you.  Then the other places e.g.
+ if (cur->content[i] == RANGE_DATUM_DEFAULT)
+ {
+ /*

will change like

+ if (cur->content == NULL)
+ {
+ /*

--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] Default Partition for Range

From
Beena Emerson
Date:
Thank you for your review Dilip and Rahila.

PFA the updated patch which is rebased to Jeevan's latest list
partition patch [1] and also handles your comments.

https://www.postgresql.org/message-id/CAOgcT0OARciE2X%2BU0rjSKp9VuC279dYcCGkc3nCWKhHQ1_m2rw%40mail.gmail.com

-- 

Beena Emerson

EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Attachment

Re: [HACKERS] Default Partition for Range

From
Beena Emerson
Date:
On Mon, Jul 3, 2017 at 8:00 PM, Dilip Kumar <dilipbalaut@gmail.com> wrote:
> On Fri, Jun 30, 2017 at 11:56 AM, Beena Emerson <memissemerson@gmail.com> wrote:
>> Hello Dilip,
>>
>> On Wed, Jun 21, 2017 at 6:27 PM, Dilip Kumar <dilipbalaut@gmail.com> wrote:
>>> On Tue, Jun 20, 2017 at 6:57 PM, Dilip Kumar <dilipbalaut@gmail.com> wrote:
>>>> This is basically crashing in RelationBuildPartitionDesc, so I think
>
>>
>> Thank you for your review and analysis.
>>
>> I have updated the patch.
>> - bound->content is set to RANGE_DATUM_DEFAULT for each of the keys
>> and not just the first one.
>> - Improve the way of handling DEFAULT bounds in RelationBuildPartitionDesc,
>>
>> There is a test for multiple column range in alter_table.sql
>
> Thanks for update patch,  After this fix segmentation fault is solved.
>
> I have some more comments.
>
> 1.
>
> lower = make_one_range_bound(key, -1, spec->lowerdatums, true);
>   upper = make_one_range_bound(key, -1, spec->upperdatums, false);
>
> + /* Default case is not handled here */
> + if (spec->is_default)
> + break;
> +
>
> I think we can move default check just above the make range bound it
> will avoid unnecessary processing.
>

Removed the check here. Default is checked beforehand.

>
> 2.
> RelationBuildPartitionDesc
> {
> ....
>
>        /*
> * If either of them has infinite element, we can't equate
> * them.  Even when both are infinite, they'd have
> * opposite signs, because only one of cur and prev is a
> * lower bound).
> */
> if (cur->content[j] != RANGE_DATUM_FINITE ||
>   prev->content[j] != RANGE_DATUM_FINITE)
> {
> is_distinct = true;
> break;
> }
> After default range partitioning patch the above comment doesn't sound
> very accurate and
> can lead to the confusion, now here we are not only considering
> infinite bounds but
> there is also a possibility that prev bound can be DEFAULT, so
> comments should clearly
> mention that.

Modified.

>
> 3.
>
> + bound->datums = datums ? (Datum *) palloc0(key->partnatts * sizeof(Datum))
> + : NULL;
>   bound->content = (RangeDatumContent *) palloc0(key->partnatts *
>     sizeof(RangeDatumContent));
>   bound->lower = lower;
>
>   i = 0;
> +
> + /* datums are NULL for default */
> + if (datums == NULL)
> + for (i = 0; i < key->partnatts; i++)
> + bound->content[i] = RANGE_DATUM_DEFAULT;
>
> To me, it will look cleaner if we keep bound->content=NULL for
> default, instead of allocating memory
> and initializing each element,  But it's just a suggestions and you
> can decide whichever way
> seems better to you.  Then the other places e.g.
> + if (cur->content[i] == RANGE_DATUM_DEFAULT)
> + {
> + /*
>
> will change like
>
> + if (cur->content == NULL)
> + {
> + /*

I disagree. We use the content value RANGE_DATUM_DEFAULT during
comparing in partition_rbound_cmp and the code will not be very
intuiative if we use NULL instead of DEFAULT.


-- 

Beena Emerson

EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company



Re: [HACKERS] Default Partition for Range

From
Beena Emerson
Date:
On Tue, Jul 4, 2017 at 4:21 PM, Rahila Syed <rahilasyed90@gmail.com> wrote:
>
> Hello Beena,
>
> Thanks for the updated patch. It passes the basic tests which I performed.
>
> Few comments:
> 1. In   check_new_partition_bound(),
>
>> /* Default case is not handled here */
>>                if (spec->is_default)
>>                    break;
> The default partition check here can be performed in common for both range
> and list partition.

Removed the check here.

>
> 2.     RANGE_DATUM_FINITE = 0,     /* actual datum stored elsewhere */
> +   RANGE_DATUM_DEFAULT,        /* default */
>
> More elaborative comment?

I am not sure what to add. Do you have something in mind?

>
> 3.  Inside make_one_range_bound(),
>
>>+
>>+   /* datums are NULL for default */
>>+   if (datums == NULL)
>>+       for (i = 0; i < key->partnatts; i++)
>>+           bound->content[i] = RANGE_DATUM_DEFAULT;
>>+
> IMO, returning bound at this stage will make code clearer as further
> processing
> does not happen for default partition.

Done.

>
> 4.
> @@ -549,6 +568,7 @@ RelationBuildPartitionDesc(Relation rel)
>                                                 sizeof(RangeDatumContent
> *));
>                     boundinfo->indexes = (int *) palloc((ndatums + 1) *
>                                                         sizeof(int));
> +                   boundinfo->default_index = -1;
> This should also be done commonly for both default list and range partition.

Removed the line here. Common allocation is done by Jeevan's patch.

>
> 5.
>               if (!spec->is_default && partdesc->nparts > 0)
>                 {
>                     ListCell   *cell;
>
>                     Assert(boundinfo &&
>                            boundinfo->strategy == PARTITION_STRATEGY_LIST &&
>                            (boundinfo->ndatums > 0 ||
>                             partition_bound_accepts_nulls(boundinfo) ||
>                             partition_bound_has_default(boundinfo)));
> The assertion condition partition_bound_has_default(boundinfo) is rendered
> useless
> because of if condition change and can be removed perhaps.

This cannot be removed.
This check is required when this code is run for a non-default
partition and default is the only existing partition.

>
> 6. I think its better to have following  regression tests coverage
>
> --Testing with inserting one NULL and other non NULL values for multicolumn
> range partitioned table.
>
> postgres=# CREATE TABLE range_parted (
> postgres(#     a int,
> postgres(#     b int
> postgres(# ) PARTITION BY RANGE (a, b);
> CREATE TABLE
> postgres=#
> postgres=# CREATE TABLE part1 (
> postgres(#     a int NOT NULL CHECK (a = 1),
> postgres(#     b int NOT NULL CHECK (b >= 1 AND b <= 10)
> postgres(# );
> CREATE TABLE
>
> postgres=# ALTER TABLE range_parted ATTACH PARTITION part1 FOR VALUES FROM
> (1, 1) TO (1, 10);
> ALTER TABLE
> postgres=# CREATE TABLE partr_def1 PARTITION OF range_parted DEFAULT;
> CREATE TABLE
>
> postgres=# insert into range_parted values (1,NULL);
> INSERT 0 1
> postgres=# insert into range_parted values (NULL,9);
> INSERT 0 1

added.





-- 

Beena Emerson

EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company



Re: [HACKERS] Default Partition for Range

From
Jeevan Ladhe
Date:
Hi Beena,

I have posted the rebased patches[1] for default list partition.
Your patch also needs a rebase.


Regards,
Jeevan Ladhe

On Fri, Jul 14, 2017 at 3:28 PM, Beena Emerson <memissemerson@gmail.com> wrote:
On Tue, Jul 4, 2017 at 4:21 PM, Rahila Syed <rahilasyed90@gmail.com> wrote:
>
> Hello Beena,
>
> Thanks for the updated patch. It passes the basic tests which I performed.
>
> Few comments:
> 1. In   check_new_partition_bound(),
>
>> /* Default case is not handled here */
>>                if (spec->is_default)
>>                    break;
> The default partition check here can be performed in common for both range
> and list partition.

Removed the check here.

>
> 2.     RANGE_DATUM_FINITE = 0,     /* actual datum stored elsewhere */
> +   RANGE_DATUM_DEFAULT,        /* default */
>
> More elaborative comment?

I am not sure what to add. Do you have something in mind?

>
> 3.  Inside make_one_range_bound(),
>
>>+
>>+   /* datums are NULL for default */
>>+   if (datums == NULL)
>>+       for (i = 0; i < key->partnatts; i++)
>>+           bound->content[i] = RANGE_DATUM_DEFAULT;
>>+
> IMO, returning bound at this stage will make code clearer as further
> processing
> does not happen for default partition.

Done.

>
> 4.
> @@ -549,6 +568,7 @@ RelationBuildPartitionDesc(Relation rel)
>                                                 sizeof(RangeDatumContent
> *));
>                     boundinfo->indexes = (int *) palloc((ndatums + 1) *
>                                                         sizeof(int));
> +                   boundinfo->default_index = -1;
> This should also be done commonly for both default list and range partition.

Removed the line here. Common allocation is done by Jeevan's patch.

>
> 5.
>               if (!spec->is_default && partdesc->nparts > 0)
>                 {
>                     ListCell   *cell;
>
>                     Assert(boundinfo &&
>                            boundinfo->strategy == PARTITION_STRATEGY_LIST &&
>                            (boundinfo->ndatums > 0 ||
>                             partition_bound_accepts_nulls(boundinfo) ||
>                             partition_bound_has_default(boundinfo)));
> The assertion condition partition_bound_has_default(boundinfo) is rendered
> useless
> because of if condition change and can be removed perhaps.

This cannot be removed.
This check is required when this code is run for a non-default
partition and default is the only existing partition.

>
> 6. I think its better to have following  regression tests coverage
>
> --Testing with inserting one NULL and other non NULL values for multicolumn
> range partitioned table.
>
> postgres=# CREATE TABLE range_parted (
> postgres(#     a int,
> postgres(#     b int
> postgres(# ) PARTITION BY RANGE (a, b);
> CREATE TABLE
> postgres=#
> postgres=# CREATE TABLE part1 (
> postgres(#     a int NOT NULL CHECK (a = 1),
> postgres(#     b int NOT NULL CHECK (b >= 1 AND b <= 10)
> postgres(# );
> CREATE TABLE
>
> postgres=# ALTER TABLE range_parted ATTACH PARTITION part1 FOR VALUES FROM
> (1, 1) TO (1, 10);
> ALTER TABLE
> postgres=# CREATE TABLE partr_def1 PARTITION OF range_parted DEFAULT;
> CREATE TABLE
>
> postgres=# insert into range_parted values (1,NULL);
> INSERT 0 1
> postgres=# insert into range_parted values (NULL,9);
> INSERT 0 1

added.





--

Beena Emerson

EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: [HACKERS] Default Partition for Range

From
Beena Emerson
Date:
On Wed, Jul 26, 2017 at 7:05 PM, Jeevan Ladhe
<jeevan.ladhe@enterprisedb.com> wrote:
> Hi Beena,
>
> I have posted the rebased patches[1] for default list partition.
> Your patch also needs a rebase.
>
> [1]
> https://www.postgresql.org/message-id/CAOgcT0OVwDu%2BbeChWb5R5s6rfKLCiWcZT5617hqu7T3GdA1hAw%40mail.gmail.com
>

Thanks for informing.
PFA the updated patch.
I have changed the numbering of enum PartitionRangeDatumKind since I
have to include DEFAULT as well. If you have better ideas, let me
know.



-- 

Beena Emerson

EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Attachment

Re: [HACKERS] Default Partition for Range

From
Robert Haas
Date:
On Mon, Jul 31, 2017 at 8:28 AM, Beena Emerson <memissemerson@gmail.com> wrote:
> Thanks for informing.
> PFA the updated patch.
> I have changed the numbering of enum PartitionRangeDatumKind since I
> have to include DEFAULT as well. If you have better ideas, let me
> know.

Why do we need to introduce PARTITION_RANGE_DATUM_DEFAULT at all?  It
seems to me that the handling of default range partitions ought to be
similar to the way a null-accepting list partition is handled -
namely, it wouldn't show up in the "datums" or "kind" array at all,
instead just showing up in PartitionBoundInfoData's default_index
field.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company



Re: [HACKERS] Default Partition for Range

From
Beena Emerson
Date:
On Fri, Aug 4, 2017 at 7:48 PM, Robert Haas <robertmhaas@gmail.com> wrote:
> On Mon, Jul 31, 2017 at 8:28 AM, Beena Emerson <memissemerson@gmail.com> wrote:
>
> Why do we need to introduce PARTITION_RANGE_DATUM_DEFAULT at all?  It
> seems to me that the handling of default range partitions ought to be
> similar to the way a null-accepting list partition is handled -
> namely, it wouldn't show up in the "datums" or "kind" array at all,
> instead just showing up in PartitionBoundInfoData's default_index
> field.
>

I have updated the patch to make it similar to the way default/null is
handled in list partition, removing the PARTITION_RANGE_DATUM_DEFAULT.
This is to be applied over v24 patches shared by Jeevan [1] which
applies on commit id 5ff3d73813ebcc3ff80be77c30b458d728951036.

The RelationBuildPartitionDesc has been modified a lot, especially the
way all_bounds, ndatums and rbounds are set.

[1] https://www.postgresql.org/message-id/CAOgcT0OVwDu%2BbeChWb5R5s6rfKLCiWcZT5617hqu7T3GdA1hAw%40mail.gmail.com

-- 

Beena Emerson

EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Attachment

Re: [HACKERS] Default Partition for Range

From
Rajkumar Raghuwanshi
Date:
On Wed, Aug 9, 2017 at 8:26 AM, Beena Emerson <memissemerson@gmail.com> wrote:
I have updated the patch to make it similar to the way default/null is
handled in list partition, removing the PARTITION_RANGE_DATUM_DEFAULT.
This is to be applied over v24 patches shared by Jeevan [1] which
applies on commit id 5ff3d73813ebcc3ff80be77c30b458d728951036.

The RelationBuildPartitionDesc has been modified a lot, especially the
way all_bounds, ndatums and rbounds are set.

[1] https://www.postgresql.org/message-id/CAOgcT0OVwDu%2BbeChWb5R5s6rfKLCiWcZT5617hqu7T3GdA1hAw%40mail.gmail.com


Hi Beena,

I have applied Jeevan's v24 patches and then your v9 patch over commit 5ff3d73813ebcc3ff80be77c30b458d728951036.
and while testing I got a server crash. below is sql to reproduce it.

postgres=# CREATE TABLE rp (a int, b int) PARTITION by range (a);
CREATE TABLE
postgres=# CREATE TABLE rp_p1 PARTITION OF rp DEFAULT partition by range(a);
CREATE TABLE
postgres=# CREATE TABLE rp_p11 PARTITION OF rp_p1 FOR VALUES FROM (1) TO (15);
CREATE TABLE
postgres=# CREATE TABLE rp_p12 PARTITION OF rp_p1 DEFAULT;
CREATE TABLE
postgres=# insert into rp select i,i from generate_series(1,15) i;
server closed the connection unexpectedly
    This probably means the server terminated abnormally
    before or while processing the request.
The connection to the server was lost. Attempting reset: Failed.

Thanks & Regards,
Rajkumar Raghuwanshi
QMG, EnterpriseDB Corporation

Re: [HACKERS] Default Partition for Range

From
Beena Emerson
Date:
Hello Rajkumar,

On Wed, Aug 9, 2017 at 12:37 PM, Rajkumar Raghuwanshi
<rajkumar.raghuwanshi@enterprisedb.com> wrote:
>
> Hi Beena,
>
> I have applied Jeevan's v24 patches and then your v9 patch over commit
> 5ff3d73813ebcc3ff80be77c30b458d728951036.
> and while testing I got a server crash. below is sql to reproduce it.
>
> postgres=# CREATE TABLE rp (a int, b int) PARTITION by range (a);
> CREATE TABLE
> postgres=# CREATE TABLE rp_p1 PARTITION OF rp DEFAULT partition by range(a);
> CREATE TABLE
> postgres=# CREATE TABLE rp_p11 PARTITION OF rp_p1 FOR VALUES FROM (1) TO
> (15);
> CREATE TABLE
> postgres=# CREATE TABLE rp_p12 PARTITION OF rp_p1 DEFAULT;
> CREATE TABLE
> postgres=# insert into rp select i,i from generate_series(1,15) i;
> server closed the connection unexpectedly
>     This probably means the server terminated abnormally
>     before or while processing the request.
> The connection to the server was lost. Attempting reset: Failed.
>

Thank you for testing. It seems I made a mistake in the assert
condition. I have corrected it in this patch.



Thank you,

Beena Emerson

EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Attachment

Re: [HACKERS] Default Partition for Range

From
Rajkumar Raghuwanshi
Date:
On Wed, Aug 9, 2017 at 1:54 PM, Beena Emerson <memissemerson@gmail.com> wrote:
Hello Rajkumar,

On Wed, Aug 9, 2017 at 12:37 PM, Rajkumar Raghuwanshi
<rajkumar.raghuwanshi@enterprisedb.com> wrote:
>
> Hi Beena,
>
> I have applied Jeevan's v24 patches and then your v9 patch over commit
> 5ff3d73813ebcc3ff80be77c30b458d728951036.
> and while testing I got a server crash. below is sql to reproduce it.
>
> postgres=# CREATE TABLE rp (a int, b int) PARTITION by range (a);
> CREATE TABLE
> postgres=# CREATE TABLE rp_p1 PARTITION OF rp DEFAULT partition by range(a);
> CREATE TABLE
> postgres=# CREATE TABLE rp_p11 PARTITION OF rp_p1 FOR VALUES FROM (1) TO
> (15);
> CREATE TABLE
> postgres=# CREATE TABLE rp_p12 PARTITION OF rp_p1 DEFAULT;
> CREATE TABLE
> postgres=# insert into rp select i,i from generate_series(1,15) i;
> server closed the connection unexpectedly
>     This probably means the server terminated abnormally
>     before or while processing the request.
> The connection to the server was lost. Attempting reset: Failed.
>

Thank you for testing. It seems I made a mistake in the assert
condition. I have corrected it in this patch.

Thanks Beena, I have tested new patch, crash got fixed now, got two observations, please check if these are expected?

--difference in the description of default partition in case of list vs range

create table lp (a int) partition by list(a);
create table lp_d partition of lp DEFAULT;
postgres=# \d+ lp_d
                                   Table "public.lp_d"
 Column |  Type   | Collation | Nullable | Default | Storage | Stats target | Description
--------+---------+-----------+----------+---------+---------+--------------+-------------
 a      | integer |           |          |         | plain   |              |
Partition of: lp DEFAULT
Partition constraint:

create table rp (a int) partition by range(a);
create table rp_d partition of rp DEFAULT;
postgres=# \d+ rp_d
                                   Table "public.rp_d"
 Column |  Type   | Collation | Nullable | Default | Storage | Stats target | Description
--------+---------+-----------+----------+---------+---------+--------------+-------------
 a      | integer |           |          |         | plain   |              |
Partition of: rp DEFAULT
Partition constraint: true

--getting warning WARNING:  skipped scanning foreign table...which is a partition of default partition...
--when adding new partition after adding default as foreign partition

CREATE EXTENSION postgres_fdw;
CREATE SERVER def_server FOREIGN DATA WRAPPER postgres_fdw OPTIONS (dbname 'postgres', port '5432', use_remote_estimate 'true');
CREATE USER MAPPING FOR PUBLIC SERVER def_server;

postgres=# CREATE TABLE frp (a int, b int) PARTITION BY RANGE(a);
CREATE TABLE
postgres=# CREATE TABLE frp_d PARTITION OF frp DEFAULT partition by RANGE(b);
CREATE TABLE
postgres=# CREATE TABLE ffrp_d_d (like frp);
CREATE TABLE
postgres=# CREATE FOREIGN TABLE ftfrp_d_d PARTITION OF frp_d DEFAULT SERVER def_server OPTIONS (TABLE_NAME 'ffrp_d_d');
CREATE FOREIGN TABLE
postgres=# CREATE TABLE frp_p2 PARTITION OF frp FOR VALUES FROM (10) TO (12);
WARNING:  skipped scanning foreign table "ftfrp_d_d" which is a partition of default partition "frp_d"
CREATE TABLE

Thanks & Regards,
Rajkumar Raghuwanshi
QMG, EnterpriseDB Corporation
 

Re: [HACKERS] Default Partition for Range

From
Robert Haas
Date:
On Wed, Aug 9, 2017 at 8:18 AM, Rajkumar Raghuwanshi
<rajkumar.raghuwanshi@enterprisedb.com> wrote:
> --difference in the description of default partition in case of list vs
> range
>
> create table lp (a int) partition by list(a);
> create table lp_d partition of lp DEFAULT;
> postgres=# \d+ lp_d
>                                    Table "public.lp_d"
>  Column |  Type   | Collation | Nullable | Default | Storage | Stats target
> | Description
> --------+---------+-----------+----------+---------+---------+--------------+-------------
>  a      | integer |           |          |         | plain   |
> |
> Partition of: lp DEFAULT
> Partition constraint:
>
> create table rp (a int) partition by range(a);
> create table rp_d partition of rp DEFAULT;
> postgres=# \d+ rp_d
>                                    Table "public.rp_d"
>  Column |  Type   | Collation | Nullable | Default | Storage | Stats target
> | Description
> --------+---------+-----------+----------+---------+---------+--------------+-------------
>  a      | integer |           |          |         | plain   |
> |
> Partition of: rp DEFAULT
> Partition constraint: true

This looks like a problem with the default list partitioning patch.  I
think "true" is what we expect to see here in both cases.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company



Re: [HACKERS] Default Partition for Range

From
Jeevan Ladhe
Date:
Hi Robert, Beena,

On Wed, Aug 9, 2017 at 5:53 PM, Robert Haas <robertmhaas@gmail.com> wrote:
On Wed, Aug 9, 2017 at 8:18 AM, Rajkumar Raghuwanshi
<rajkumar.raghuwanshi@enterprisedb.com> wrote:
> --difference in the description of default partition in case of list vs
> range
>
> create table lp (a int) partition by list(a);
> create table lp_d partition of lp DEFAULT;
> postgres=# \d+ lp_d
>                                    Table "public.lp_d"
>  Column |  Type   | Collation | Nullable | Default | Storage | Stats target
> | Description
> --------+---------+-----------+----------+---------+---------+--------------+-------------
>  a      | integer |           |          |         | plain   |
> |
> Partition of: lp DEFAULT
> Partition constraint:
>
> create table rp (a int) partition by range(a);
> create table rp_d partition of rp DEFAULT;
> postgres=# \d+ rp_d
>                                    Table "public.rp_d"
>  Column |  Type   | Collation | Nullable | Default | Storage | Stats target
> | Description
> --------+---------+-----------+----------+---------+---------+--------------+-------------
>  a      | integer |           |          |         | plain   |
> |
> Partition of: rp DEFAULT
> Partition constraint: true

This looks like a problem with the default list partitioning patch.  I
think "true" is what we expect to see here in both cases.

In case of list partition if there is only default partition, then there is no
specific value set that can be excluded from default partition, so in
such a case DEFAULT partition for a list partitioned table simply will
not have any constraint on it, and hence while the describe output is
printed it does not have any constraints to be printed.

Whereas, in case of default partition for a range partitioned table, in
get_qual_for_range() it creates true boolean expression if default is the
only partition. Here is the relevent code from Beena's patch:

+ else /* default is the only partition */
+ result = list_make1(makeBoolConst(true, false));

I think, it is unnecessary to create a constraint that is going to be always
true. Instead, if we have no constraints we can avoid some extra cpu
cycles which would otherwise be wasted in evaluating an expression
that is going to be always true.

Having said this, I see a point that in such a case we should have
some neat meaningful content to be printed for "Partition constraint: ".
I think we can address this when we construct describe output string.

Some ideas that come to my mind are:
Partition constraint: NIL
Partition constraint: no constraints
No partition constraint.
Partition constraint: true

Please let me know your thoughts.

Regards,
Jeevan Ladhe  

Re: [HACKERS] Default Partition for Range

From
Robert Haas
Date:
On Thu, Aug 10, 2017 at 6:56 AM, Jeevan Ladhe
<jeevan.ladhe@enterprisedb.com> wrote:
>> This looks like a problem with the default list partitioning patch.  I
>> think "true" is what we expect to see here in both cases.
>
> In case of list partition if there is only default partition, then there is
> no
> specific value set that can be excluded from default partition, so in
> such a case DEFAULT partition for a list partitioned table simply will
> not have any constraint on it, and hence while the describe output is
> printed it does not have any constraints to be printed.
>
> Whereas, in case of default partition for a range partitioned table, in
> get_qual_for_range() it creates true boolean expression if default is the
> only partition. Here is the relevent code from Beena's patch:
>
> + else /* default is the only partition */
> + result = list_make1(makeBoolConst(true, false));
>
> I think, it is unnecessary to create a constraint that is going to be always
> true. Instead, if we have no constraints we can avoid some extra cpu
> cycles which would otherwise be wasted in evaluating an expression
> that is going to be always true.

That's a reasonable argument.  I'm not sure whether having no
partition constraint at all is likely to be a source of bugs, but
certainly, not needing to check it is appealing.

> Having said this, I see a point that in such a case we should have
> some neat meaningful content to be printed for "Partition constraint: ".
> I think we can address this when we construct describe output string.
>
> Some ideas that come to my mind are:
> Partition constraint: NIL
> Partition constraint: no constraints
> No partition constraint.
> Partition constraint: true
>
> Please let me know your thoughts.

I like "No partition constraint." of those options.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company



Re: [HACKERS] Default Partition for Range

From
Beena Emerson
Date:
Hello,

PFA the updated patch which returns NULL instead of true when the
default partition has no constraints and also have modified the output
as discussed above.

This applies over v24 patch [1] of default list partition. I will
rebase over the next version when it is updated.

[1] https://www.postgresql.org/message-id/CAOgcT0OVwDu%2BbeChWb5R5s6rfKLCiWcZT5617hqu7T3GdA1hAw%40mail.gmail.com
-- 

Beena Emerson

EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Attachment

Re: [HACKERS] Default Partition for Range

From
Beena Emerson
Date:
PFA the patch rebased over v25 patches of default list partition [1]

[1] https://www.postgresql.org/message-id/CAOgcT0NwqnavYtu-QM-DAZ6N%3DwTiqKgy83WwtO2x94LSLZ1-Sw%40mail.gmail.com




-- 

Beena Emerson

EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Attachment

Re: [HACKERS] Default Partition for Range

From
Jeevan Ladhe
Date:

Hi Beena,

On Thu, Aug 17, 2017 at 4:59 PM, Beena Emerson <memissemerson@gmail.com> wrote:
PFA the patch rebased over v25 patches of default list partition [1]

Thanks for rebasing.

Range partition review:

1.
There are lot of changes in RelationBuildPartitionDesc(). It was hard to
understand why these changes are needed for default partition. I did not find
any explanation for the same, may be I am missing some discussion? Only
when I looked into it carefully I could understand that these changes are
nothing but optimization for identifying the distinct bounds.
I think it is better to keep the patch for code optimisation/simplification in a
separate patch. I have separated the patch(0001) in attached patches for this
purpose.

2.
+ * For default partition, it returns the negation of the constraints of all
+ * the other partitions.
+ *
+ * If we end up with an empty result list, we return NULL.

We can rephrase the comment as below:

"For default partition, function returns the negation of the constraints of all
the other partitions. If default is the only partition then returns NULL."

3.
@@ -2396,6 +2456,8 @@ make_one_range_bound(PartitionKey key, int index, List *datums, bool lower)
    ListCell   *lc;
    int         i;

+   Assert(datums != NULL);
+
    bound = (PartitionRangeBound *) palloc0(sizeof(PartitionRangeBound));
    bound->index = index;
    bound->datums = (Datum *) palloc0(key->partnatts * sizeof(Datum));

I am not really convinced, why should we have above Assert.

4.
 static List *
-get_qual_for_range(PartitionKey key, PartitionBoundSpec *spec)
+get_qual_for_range(Relation parent, PartitionBoundSpec *spec,
+                  bool for_default)
 {

The addition and the way flag ‘for_default’ is being used is very confusing.
At this moment I am not able to think about a alternate solution to the
way you have used this flag. But will try and see if I can come up with
an alternate approach.

I still need to look at the test, and need to do some manual testing. Will
update here with any findings.

Patches:
0001: Refactoring/simplification of code in RelationBuildPartitionDesc(),
0002: implementation of default range partition by Beena.

Regards,
Jeevan 
Attachment

Re: [HACKERS] Default Partition for Range

From
Beena Emerson
Date:
Hi Jeevan,

On Tue, Aug 22, 2017 at 7:53 AM, Jeevan Ladhe
<jeevan.ladhe@enterprisedb.com> wrote:
>
> Hi Beena,
>
> On Thu, Aug 17, 2017 at 4:59 PM, Beena Emerson <memissemerson@gmail.com>
> wrote:
>>
>> PFA the patch rebased over v25 patches of default list partition [1]
>
>
> Thanks for rebasing.
>
> Range partition review:

Thank you for your review.

>
> 3.
> @@ -2396,6 +2456,8 @@ make_one_range_bound(PartitionKey key, int index, List
> *datums, bool lower)
>     ListCell   *lc;
>     int         i;
>
> +   Assert(datums != NULL);
> +
>     bound = (PartitionRangeBound *) palloc0(sizeof(PartitionRangeBound));
>     bound->index = index;
>     bound->datums = (Datum *) palloc0(key->partnatts * sizeof(Datum));
>
> I am not really convinced, why should we have above Assert.

The function should never be called for default partition where datums = NULL.

>
> 4.
>  static List *
> -get_qual_for_range(PartitionKey key, PartitionBoundSpec *spec)
> +get_qual_for_range(Relation parent, PartitionBoundSpec *spec,
> +                  bool for_default)
>  {
>
> The addition and the way flag ‘for_default’ is being used is very confusing.
> At this moment I am not able to think about a alternate solution to the
> way you have used this flag. But will try and see if I can come up with
> an alternate approach.

Well, we need to skip the get_range_nulltest while collecting the qual
of other partition to make one for default. We could skip this flag
and remove the NullTest from the qual returned for each partition but
I am not sure if thats a good way to go about it.


--

Beena Emerson

EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company



Re: [HACKERS] Default Partition for Range

From
Beena Emerson
Date:
On Tue, Aug 22, 2017 at 4:20 PM, Beena Emerson <memissemerson@gmail.com> wrote:
> Hi Jeevan,
>
> On Tue, Aug 22, 2017 at 7:53 AM, Jeevan Ladhe
> <jeevan.ladhe@enterprisedb.com> wrote:
>>
>>
>> 4.
>>  static List *
>> -get_qual_for_range(PartitionKey key, PartitionBoundSpec *spec)
>> +get_qual_for_range(Relation parent, PartitionBoundSpec *spec,
>> +                  bool for_default)
>>  {
>>
>> The addition and the way flag ‘for_default’ is being used is very confusing.
>> At this moment I am not able to think about a alternate solution to the
>> way you have used this flag. But will try and see if I can come up with
>> an alternate approach.
>
> Well, we need to skip the get_range_nulltest while collecting the qual
> of other partition to make one for default. We could skip this flag
> and remove the NullTest from the qual returned for each partition but
> I am not sure if thats a good way to go about it.
>
>

Please find attached a patch which removes the for_default flag from
the get_qual_for_range function. This patch is just to show an idea
and should be applied over my earlier patch. There could be a better
way to do this. Let me know your opinion.


--

Beena Emerson

EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Attachment

Re: [HACKERS] Default Partition for Range

From
Jeevan Ladhe
Date:
Hi Beena,


On Tue, Aug 22, 2017 at 5:22 PM, Beena Emerson <memissemerson@gmail.com> wrote:
On Tue, Aug 22, 2017 at 4:20 PM, Beena Emerson <memissemerson@gmail.com> wrote:
> Hi Jeevan,
>
> On Tue, Aug 22, 2017 at 7:53 AM, Jeevan Ladhe
> <jeevan.ladhe@enterprisedb.com> wrote:
>>
>>
>> 4.
>>  static List *
>> -get_qual_for_range(PartitionKey key, PartitionBoundSpec *spec)
>> +get_qual_for_range(Relation parent, PartitionBoundSpec *spec,
>> +                  bool for_default)
>>  {
>>
>> The addition and the way flag ‘for_default’ is being used is very confusing.
>> At this moment I am not able to think about a alternate solution to the
>> way you have used this flag. But will try and see if I can come up with
>> an alternate approach.
>
> Well, we need to skip the get_range_nulltest while collecting the qual
> of other partition to make one for default. We could skip this flag
> and remove the NullTest from the qual returned for each partition but
> I am not sure if thats a good way to go about it.
>
>

Please find attached a patch which removes the for_default flag from
the get_qual_for_range function. This patch is just to show an idea
and should be applied over my earlier patch. There could be a better
way to do this. Let me know your opinion.


+
+ foreach (lc, list_tmp)
+ {
+ Node *n = (Node *) lfirst(lc);
+
+ if (IsA(n, NullTest))
+ {
+ list_delete(part_qual, n);
+ count++;
+ }
+ }
+ 

I think its an unnecessary overhead to have the nulltest prepared first
and then search for it and remove it from the partition qual. This is very ugly.
I think the original idea is still better compared to this. May be we can rename
the 'for_default' flag to something like 'part_of_default_qual'.

Also, as discussed offline I will merge this with default list partitioning patch.

Regards,
Jeevan Ladhe

Re: [HACKERS] Default Partition for Range

From
Beena Emerson
Date:
Hello,

On Thu, Aug 24, 2017 at 3:08 PM, Jeevan Ladhe
<jeevan.ladhe@enterprisedb.com> wrote:
> Hi Beena,
>
>
> On Tue, Aug 22, 2017 at 5:22 PM, Beena Emerson <memissemerson@gmail.com>
> wrote:
>>
>> On Tue, Aug 22, 2017 at 4:20 PM, Beena Emerson <memissemerson@gmail.com>
>> wrote:
>> > Hi Jeevan,
>> >
>> > On Tue, Aug 22, 2017 at 7:53 AM, Jeevan Ladhe
>> > <jeevan.ladhe@enterprisedb.com> wrote:
>> >>
>> >>
>> >> 4.
>> >>  static List *
>> >> -get_qual_for_range(PartitionKey key, PartitionBoundSpec *spec)
>> >> +get_qual_for_range(Relation parent, PartitionBoundSpec *spec,
>> >> +                  bool for_default)
>> >>  {
>> >>
>> >> The addition and the way flag ‘for_default’ is being used is very
>> >> confusing.
>> >> At this moment I am not able to think about a alternate solution to the
>> >> way you have used this flag. But will try and see if I can come up with
>> >> an alternate approach.
>> >
>> > Well, we need to skip the get_range_nulltest while collecting the qual
>> > of other partition to make one for default. We could skip this flag
>> > and remove the NullTest from the qual returned for each partition but
>> > I am not sure if thats a good way to go about it.
>> >
>> >
>>
>> Please find attached a patch which removes the for_default flag from
>> the get_qual_for_range function. This patch is just to show an idea
>> and should be applied over my earlier patch. There could be a better
>> way to do this. Let me know your opinion.
>>
>
> +
> + foreach (lc, list_tmp)
> + {
> + Node *n = (Node *) lfirst(lc);
> +
> + if (IsA(n, NullTest))
> + {
> + list_delete(part_qual, n);
> + count++;
> + }
> + }
> +
>
> I think its an unnecessary overhead to have the nulltest prepared first
> and then search for it and remove it from the partition qual. This is very
> ugly.

Yes, I felt the same but just put it out there to see if someone can
improve on this.

> I think the original idea is still better compared to this. May be we can
> rename
> the 'for_default' flag to something like 'part_of_default_qual'.
>

Ya. I think that would work.



--

Beena Emerson

EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company



Re: [HACKERS] Default Partition for Range

From
Beena Emerson
Date:
Hello all,

This is to inform all that this patch has been merged with default
list partition patch [1]. There will be no further updates here. The
status of this will be updated on the commitfest according to progres
on that thread.


[1] https://www.postgresql.org/message-id/CAOgcT0ONgwajdtkoq%2BAuYkdTPY9cLWWLjxt_k4SXue3eieAr%2Bg%40mail.gmail.com

Thank you,

Beena Emerson

EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company