Thread: Is there a way to SubPartition?

Is there a way to SubPartition?

From
Jerry Champlin
Date:
Is there a way to use multi-level inheritance to achieve sub
partitioning that the query optimizer will recognize?  With our current
application design, we would need a partition for every other day  for
18 months which will not perform well.  The reason we need so many
partitions is that we can't afford to vacuum the active partition (750MM
inserts + updates per day is the performance requirement for 12 months
out).  After it's a day old, there are no  longer any updates or inserts
and we can vacuum  it at that point.  If multi-level partitioning
worked, we could solve this problem without changing our code.  Ideas?

-Jerry



Re: Is there a way to SubPartition?

From
Tom Lane
Date:
Jerry Champlin <jchamplin@absolute-performance.com> writes:
> Is there a way to use multi-level inheritance to achieve sub
> partitioning that the query optimizer will recognize?

No, I don't think so.  How would that make things any better anyway?
You're still going to end up with the same very large number of
partitions.

            regards, tom lane

Re: Is there a way to SubPartition?

From
Matthew Wakeling
Date:
On Wed, 27 Aug 2008, Jerry Champlin wrote:
> After it's a day old, there are no longer any updates or inserts and we
> can vacuum it at that point.

A pattern that has worked very well for other people is to have two
separate tables (or partitions). One contains today's data, and the other
contains historic data that is no longer updated. Once a day, transfer the
data between the partitions, and the historic data partition will not need
vacuuming.

Some changes to your code will be needed however.

Matthew

--
Vacuums are nothings. We only mention them to let them know we know
they're there.

Re: Is there a way to SubPartition?

From
"Jerry Champlin"
Date:
If it were implemented in such a way that when the top level pruning
happens, a set of 3 sub partitions is selected from say 18 total and then at
the next level is selects the 3 matching sub partitions from each matched
group of 30 then you are only looking at 18+3*30 = 108 instead of 548 checks
to evaluate <example assumes monthly first level partitioning and daily sub
partitioning>.  If this is not supported, then we will need to solve the
problem a different way - probably weekly partitions and refactor the code
to decrease updates by at least an order of magnitude.  While we are in the
process of doing this, is there a way to make updates faster?  Postgresql is
spending a lot of CPU cycles for each HOT update.  We have
synchronous_commit turned off, commit siblings set to 5, commit_delay set to
50,000.  With synchronous_commit off does it make any sense to be grouping
commits?  Buffers written by the bgwriter vs checkpoint is 6 to 1.  Buffers
written by clients vs buffers by checkpoint is 1 to 6.  Is there anything
obvious here?

-Jerry


-----Original Message-----
From: Tom Lane [mailto:tgl@sss.pgh.pa.us]
Sent: Wednesday, August 27, 2008 8:02 AM
To: Jerry Champlin
Cc: pgsql-performance@postgresql.org
Subject: Re: [PERFORM] Is there a way to SubPartition?

Jerry Champlin <jchamplin@absolute-performance.com> writes:
> Is there a way to use multi-level inheritance to achieve sub
> partitioning that the query optimizer will recognize?

No, I don't think so.  How would that make things any better anyway?
You're still going to end up with the same very large number of
partitions.

            regards, tom lane



Re: Is there a way to SubPartition?

From
Alvaro Herrera
Date:
Jerry Champlin wrote:
> If it were implemented in such a way that when the top level pruning
> happens, a set of 3 sub partitions is selected from say 18 total and then at
> the next level is selects the 3 matching sub partitions from each matched
> group of 30 then you are only looking at 18+3*30 = 108 instead of 548 checks
> to evaluate <example assumes monthly first level partitioning and daily sub
> partitioning>.  If this is not supported, then we will need to solve the
> problem a different way - probably weekly partitions and refactor the code
> to decrease updates by at least an order of magnitude.  While we are in the
> process of doing this, is there a way to make updates faster?  Postgresql is
> spending a lot of CPU cycles for each HOT update.  We have
> synchronous_commit turned off, commit siblings set to 5, commit_delay set to
> 50,000.

Perhaps you do not realize this, but this is an exciting report to read.
Not many years ago, this kind of system would have been unthinkable.
We've now tuned the system so that people is starting to consider it,
and for a lot of people it is working.

--
Alvaro Herrera                                http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

Re: Is there a way to SubPartition?

From
Tom Lane
Date:
"Jerry Champlin" <jchamplin@absolute-performance.com> writes:
> We have synchronous_commit turned off, commit siblings set to 5,
> commit_delay set to 50,000.  With synchronous_commit off does it make
> any sense to be grouping commits?

No.  In fact commit_delay is a total no-op in that mode.  If it were
doing anything I think you'd have found that to be a counterproductively
large setting ...

            regards, tom lane