Thread: Partition: use triggers instead of rules

Folks,

Best practices for partitioning so far have shown that TRIGGERs are
better than RULEs for most cases.  Please find attached a patch which
reflects this.

Thanks to Robert Treat for help putting this together :)

Cheers,
David.
--
David Fetter <david@fetter.org> http://fetter.org/
Phone: +1 415 235 3778  AIM: dfetter666  Yahoo!: dfetter
Skype: davidfetter      XMPP: david.fetter@gmail.com

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Wed, 28 Nov 2007 12:26:15 -0800
David Fetter <david@fetter.org> wrote:

> Folks,
> 
> Best practices for partitioning so far have shown that TRIGGERs are
> better than RULEs for most cases.  Please find attached a patch which
> reflects this.
> 
> Thanks to Robert Treat for help putting this together :)
> 
> Cheers,
> David.

+1

Joshua D. Drake

- -- 

      === The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564   24x7/Emergency: +1.800.492.2240
PostgreSQL solutions since 1997  http://www.commandprompt.com/
            UNIQUE NOT NULL
Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
PostgreSQL Replication: http://www.commandprompt.com/products/

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFHTdHoATb/zqfZUUQRAnpWAJ9xHqMnNorANuS9r8Hm90tn2bmZIwCglwlq
D7Q1C5vcbvQ/JiqVcv0wCho=
=NcWZ
-----END PGP SIGNATURE-----

On Wed, Nov 28, 2007 at 12:39:04PM -0800, Joshua D. Drake wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On Wed, 28 Nov 2007 12:26:15 -0800
> David Fetter <david@fetter.org> wrote:
>
> > Folks,
> >
> > Best practices for partitioning so far have shown that TRIGGERs are
> > better than RULEs for most cases.  Please find attached a patch which
> > reflects this.
> >
> > Thanks to Robert Treat for help putting this together :)
> >
> > Cheers,
> > David.
>
> +1
>
> Joshua D. Drake

Per Robert, I've also dropped the UNION partitioning suggestion as
it's pretty useless.

Cheers,
David.
--
David Fetter <david@fetter.org> http://fetter.org/
Phone: +1 415 235 3778  AIM: dfetter666  Yahoo!: dfetter
Skype: davidfetter      XMPP: david.fetter@gmail.com

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

On Wed, Nov 28, 2007 at 12:41:20PM -0800, David Fetter wrote:
> On Wed, Nov 28, 2007 at 12:39:04PM -0800, Joshua D. Drake wrote:
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA1
> >
> > On Wed, 28 Nov 2007 12:26:15 -0800
> > David Fetter <david@fetter.org> wrote:
> >
> > > Folks,
> > >
> > > Best practices for partitioning so far have shown that TRIGGERs are
> > > better than RULEs for most cases.  Please find attached a patch which
> > > reflects this.
> > >
> > > Thanks to Robert Treat for help putting this together :)
> > >
> > > Cheers,
> > > David.
> >
> > +1
> >
> > Joshua D. Drake
>
> Per Robert, I've also dropped the UNION partitioning suggestion as
> it's pretty useless.

Oops.  Patch including *both* changes attached this time.

Cheers,
David.
--
David Fetter <david@fetter.org> http://fetter.org/
Phone: +1 415 235 3778  AIM: dfetter666  Yahoo!: dfetter
Skype: davidfetter      XMPP: david.fetter@gmail.com

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

David Fetter <david@fetter.org> writes:
> Best practices for partitioning so far have shown that TRIGGERs are
> better than RULEs for most cases.  Please find attached a patch which
> reflects this.

Entirely removing the example of how to do it with rules doesn't
seem like a good idea.

            regards, tom lane

On Wed, Nov 28, 2007 at 03:53:04PM -0500, Tom Lane wrote:
> David Fetter <david@fetter.org> writes:
> > Best practices for partitioning so far have shown that TRIGGERs
> > are better than RULEs for most cases.  Please find attached a
> > patch which reflects this.
>
> Entirely removing the example of how to do it with rules doesn't
> seem like a good idea.

It does to me.  I haven't found a case yet where rules worked even as
well as triggers.

Cheers,
David.
--
David Fetter <david@fetter.org> http://fetter.org/
Phone: +1 415 235 3778  AIM: dfetter666  Yahoo!: dfetter
Skype: davidfetter      XMPP: david.fetter@gmail.com

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Wed, 28 Nov 2007 12:57:10 -0800
David Fetter <david@fetter.org> wrote:

> On Wed, Nov 28, 2007 at 03:53:04PM -0500, Tom Lane wrote:
> > David Fetter <david@fetter.org> writes:
> > > Best practices for partitioning so far have shown that TRIGGERs
> > > are better than RULEs for most cases.  Please find attached a
> > > patch which reflects this.
> > 
> > Entirely removing the example of how to do it with rules doesn't
> > seem like a good idea.
> 
> It does to me.  I haven't found a case yet where rules worked even as
> well as triggers.

I have yet to see a place that rules are appropriate at all with
partitioning. The only half reasonable argument to their existence with
it is that they are easier for someone with an extremely simple
partitioning configuration.

However rules are so deficient on anything except the absolute most
basic example of partitioning that I think we are doing our new
partitioning users a disservice by stating anything about rules except
in passing.

Sincerely,

Joshua D. Drake

> 
> Cheers,
> David.


- -- 

      === The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564   24x7/Emergency: +1.800.492.2240
PostgreSQL solutions since 1997  http://www.commandprompt.com/
            UNIQUE NOT NULL
Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
PostgreSQL Replication: http://www.commandprompt.com/products/

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFHTdeuATb/zqfZUUQRAvf3AJ96e4YynZ/rYv6egG+w+85Ms+ihogCgjIlh
6xWR0nzceoJYCiQl+ffFTK8=
=XREP
-----END PGP SIGNATURE-----

David Fetter <david@fetter.org> writes:
> On Wed, Nov 28, 2007 at 03:53:04PM -0500, Tom Lane wrote:
>> Entirely removing the example of how to do it with rules doesn't
>> seem like a good idea.

> It does to me.  I haven't found a case yet where rules worked even as
> well as triggers.

I don't have a problem with emphasizing triggers as the preferred
solution, but we should keep the older example, if only because people
are going to see DB schemas that use that approach, and they won't
understand what's going on (or realize they could convert) if they've
not seen an example.

            regards, tom lane

On Wed, Nov 28, 2007 at 05:17:38PM -0500, Tom Lane wrote:
> David Fetter <david@fetter.org> writes:
> > On Wed, Nov 28, 2007 at 03:53:04PM -0500, Tom Lane wrote:
> >> Entirely removing the example of how to do it with rules doesn't
> >> seem like a good idea.
>
> > It does to me.  I haven't found a case yet where rules worked even
> > as well as triggers.
>
> I don't have a problem with emphasizing triggers as the preferred
> solution, but we should keep the older example, if only because
> people are going to see DB schemas that use that approach, and they
> won't understand what's going on (or realize they could convert) if
> they've not seen an example.

Greg Sabino Mullane managed to contrive an example where RULEs might
conceivably be the least-bad way to do this, that being a machine
where no PLs may be installed.

Telling people how to do this is not *quite* as bad as describing how
to do EAV, but it's pretty close.

Cheers,
David.
--
David Fetter <david@fetter.org> http://fetter.org/
Phone: +1 415 235 3778  AIM: dfetter666  Yahoo!: dfetter
Skype: davidfetter      XMPP: david.fetter@gmail.com

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

David Fetter wrote:

> Greg Sabino Mullane managed to contrive an example where RULEs might
> conceivably be the least-bad way to do this, that being a machine
> where no PLs may be installed.

Perhaps this just means we should consider installing plpgsql by
default.

--
Alvaro Herrera                  http://www.amazon.com/gp/registry/5ZYLFMCVHXC
"There was no reply" (Kernel Traffic)

On Wed, Nov 28, 2007 at 10:06:01PM -0300, Alvaro Herrera wrote:
> David Fetter wrote:
>
> > Greg Sabino Mullane managed to contrive an example where RULEs
> > might conceivably be the least-bad way to do this, that being a
> > machine where no PLs may be installed.
>
> Perhaps this just means we should consider installing plpgsql by
> default.

I'm all for that :)

Cheers,
David.
--
David Fetter <david@fetter.org> http://fetter.org/
Phone: +1 415 235 3778  AIM: dfetter666  Yahoo!: dfetter
Skype: davidfetter      XMPP: david.fetter@gmail.com

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

On Nov 28, 2007 3:53 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Entirely removing the example of how to do it with rules doesn't
> seem like a good idea.

Agreed.

--
Jonah H. Harris, Sr. Software Architect | phone: 732.331.1324
EnterpriseDB Corporation                | fax: 732.331.1301
499 Thornall Street, 2nd Floor          | jonah.harris@enterprisedb.com
Edison, NJ 08837                        | http://www.enterprisedb.com/

On Wed, Nov 28, 2007 at 09:58:26PM -0500, Jonah H. Harris wrote:
> On Nov 28, 2007 3:53 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> > Entirely removing the example of how to do it with rules doesn't
> > seem like a good idea.
>
> Agreed.

Do you have an example of one use case where using RULEs rather than
TRIGGERs is a good idea?

Cheers,
David.
--
David Fetter <david@fetter.org> http://fetter.org/
Phone: +1 415 235 3778  AIM: dfetter666  Yahoo!: dfetter
Skype: davidfetter      XMPP: david.fetter@gmail.com

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

David Fetter <david@fetter.org> writes:
> On Wed, Nov 28, 2007 at 09:58:26PM -0500, Jonah H. Harris wrote:
>> On Nov 28, 2007 3:53 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>>> Entirely removing the example of how to do it with rules doesn't
>>> seem like a good idea.
>>
>> Agreed.

> Do you have an example of one use case where using RULEs rather than
> TRIGGERs is a good idea?

The argument I made for keeping the example around is not dependent on
the assumption that using a rule is a good idea.  It's dependent on the
established fact that we have recommended that in prior releases, and
therefore people are going to be seeing that construct in real
databases.

            regards, tom lane

On Wed, 28 Nov 2007, Alvaro Herrera wrote:

> David Fetter wrote:
>
> > Greg Sabino Mullane managed to contrive an example where RULEs might
> > conceivably be the least-bad way to do this, that being a machine
> > where no PLs may be installed.
>
> Perhaps this just means we should consider installing plpgsql by
> default.

I have run into this myself, and a patch that I contributed (which made it
in to 8.3) made it possible for a database owner to create trusted
languages from templates in the default configuration.  Which means that
if an admin wants to prevent usage of the language, they can revoke the
right to create it, but db owners still "opt-in" to any languages they
want.

On Thu, Nov 29, 2007 at 12:55:53AM -0500, Tom Lane wrote:
> David Fetter <david@fetter.org> writes:
> > On Wed, Nov 28, 2007 at 09:58:26PM -0500, Jonah H. Harris wrote:
> >> On Nov 28, 2007 3:53 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> >>> Entirely removing the example of how to do it with rules doesn't
> >>> seem like a good idea.
> >>
> >> Agreed.
>
> > Do you have an example of one use case where using RULEs rather
> > than TRIGGERs is a good idea?
>
> The argument I made for keeping the example around is not dependent
> on the assumption that using a rule is a good idea.  It's dependent
> on the established fact that we have recommended that in prior
> releases, and therefore people are going to be seeing that construct
> in real databases.

We've corrected mistakes in the manual before :)

Cheers,
David.
--
David Fetter <david@fetter.org> http://fetter.org/
Phone: +1 415 235 3778  AIM: dfetter666  Yahoo!: dfetter
Skype: davidfetter      XMPP: david.fetter@gmail.com

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Thu, 29 Nov 2007 00:55:53 -0500
Tom Lane <tgl@sss.pgh.pa.us> wrote:

> David Fetter <david@fetter.org> writes:
> > On Wed, Nov 28, 2007 at 09:58:26PM -0500, Jonah H. Harris wrote:
> >> On Nov 28, 2007 3:53 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> >>> Entirely removing the example of how to do it with rules doesn't
> >>> seem like a good idea.
> >> 
> >> Agreed.
> 
> > Do you have an example of one use case where using RULEs rather than
> > TRIGGERs is a good idea?
> 
> The argument I made for keeping the example around is not dependent on
> the assumption that using a rule is a good idea.  It's dependent on
> the established fact that we have recommended that in prior releases,
> and therefore people are going to be seeing that construct in real
> databases.

And they could refer back to the older version of the documentation for
it. In fact, we should mention that in the patch:

<note><para>If you have an partitioning setup that uses rules please
refer to the 8.2 documentation on partitioning</para></note>

Sincerely,


Joshua D. Drake


> 
>             regards, tom lane
> 
> ---------------------------(end of
> broadcast)--------------------------- TIP 4: Have you searched our
> list archives?
> 
>                http://archives.postgresql.org
> 


- -- 

      === The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564   24x7/Emergency: +1.800.492.2240
PostgreSQL solutions since 1997  http://www.commandprompt.com/
            UNIQUE NOT NULL
Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
PostgreSQL Replication: http://www.commandprompt.com/products/

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFHTl84ATb/zqfZUUQRAtscAJ9h77nDNJ3ZggWgocXDQaFE/S998gCcDtiV
wKH3BejsoL0fR5D3KHhKaEc=
=zpPv
-----END PGP SIGNATURE-----

NikhilS wrote:
>> The argument I made for keeping the example around is not
> dependent on
>> the assumption that using a rule is a good idea.  It's dependent on
>>  the established fact that we have recommended that in prior
>> releases, and therefore people are going to be seeing that
>> construct in real databases.
>
> And they could refer back to the older version of the documentation
> for it. In fact, we should mention that in the patch:
>
> <note><para>If you have a partitioning setup that uses rules please
> refer to the 8.2 documentation on partitioning</para></note>
>
> +1

I would also add another sentence about *why* the recommendation was
changed. We have one rule-based setup here, and it has been working
flawlessly for us,... so personally I don't even know the reasons.

Best Regards
Michael Paesold

Michael Paesold wrote:
> NikhilS wrote:

>> <note><para>If you have a partitioning setup that uses rules please
>> refer to the 8.2 documentation on partitioning</para></note>
>>
>> +1
>
> I would also add another sentence about *why* the recommendation was
> changed. We have one rule-based setup here, and it has been working
> flawlessly for us,... so personally I don't even know the reasons.
>

Rules are extremely slow in comparisons and not anywhere near as
flexible. As I said up post yesterday... they work well in the basic
partitioning configuration but anything else they are extremely deficient.

Joshua D. Drake

> Best Regards
> Michael Paesold
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 1: if posting/reading through Usenet, please send an appropriate
>       subscribe-nomail command to majordomo@postgresql.org so that your
>       message can get through to the mailing list cleanly
>

Joshua D. Drake wrote:
> Michael Paesold wrote:
>> I would also add another sentence about *why* the recommendation was
>> changed. We have one rule-based setup here, and it has been working
>> flawlessly for us,... so personally I don't even know the reasons.
>>
>
> Rules are extremely slow in comparisons and not anywhere near as
> flexible. As I said up post yesterday... they work well in the basic
> partitioning configuration but anything else they are extremely deficient.

Ah, thanks for that summary. How do they compare to triggers dynamically
EXECUTEing the inserts? Is that a better solution, or should one really
just use the IF ... ELSIF ... ELSIF ... ELSE pattern as suggested in the
new docs? (Which means one has to re-create the complete trigger each
time a partition is added.)

Best Regards
Michael Paesold

"Joshua D. Drake" <jd@commandprompt.com> writes:
> Rules are extremely slow in comparisons and not anywhere near as
> flexible. As I said up post yesterday... they work well in the basic
> partitioning configuration but anything else they are extremely deficient.

I think that the above claim is exceedingly narrow-minded.  A trigger
will probably beat a rule for inserts/updates involving a small number
of rows.  For large numbers of rows, like an INSERT/SELECT from another
large table, the rule is likely to win, because its overhead is paid
once per query not once per row.  Also, if you implement the trigger
with an EXECUTE (forcing a planning cycle) intead of hard-coded
commands, the speed advantage becomes even more dubious.

            regards, tom lane

Tom Lane wrote:
> "Joshua D. Drake" <jd@commandprompt.com> writes:
>> Rules are extremely slow in comparisons and not anywhere near as
>> flexible. As I said up post yesterday... they work well in the basic
>> partitioning configuration but anything else they are extremely deficient.
>
> I think that the above claim is exceedingly narrow-minded.

We are talking about partitioning. It is supposed to be narrow-minded.

>  A trigger
> will probably beat a rule for inserts/updates involving a small number
> of rows.

Which is exactly what partitioning is doing.

  For large numbers of rows, like an INSERT/SELECT from another
> large table, the rule is likely to win, because its overhead is paid
> once per query not once per row.  Also, if you implement the trigger
> with an EXECUTE (forcing a planning cycle) intead of hard-coded
> commands, the speed advantage becomes even more dubious.

Not for partitioning. Although I agree with your sentiments for normal
operation.

Sincerely,

Joshua D. Drake

Joshua D. Drake wrote:
> Tom Lane wrote:
> > "Joshua D. Drake" <jd@commandprompt.com> writes:
> >> Rules are extremely slow in comparisons and not anywhere near as
> >> flexible. As I said up post yesterday... they work well in the basic
> >> partitioning configuration but anything else they are extremely deficient.
> >
> > I think that the above claim is exceedingly narrow-minded.
>
> We are talking about partitioning. It is supposed to be narrow-minded.

Sure, but look at all the confusion we have had just on this list about
it.  We had better state why triggers should be used in place of rules
_for_ _partitioning_ or that confusion will continue.

--
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://postgres.enterprisedb.com

  + If your life is a hard drive, Christ can be your backup. +

Bruce Momjian wrote:
> Joshua D. Drake wrote:
>> Tom Lane wrote:
>>> "Joshua D. Drake" <jd@commandprompt.com> writes:
>>>> Rules are extremely slow in comparisons and not anywhere near as
>>>> flexible. As I said up post yesterday... they work well in the basic
>>>> partitioning configuration but anything else they are extremely deficient.
>>> I think that the above claim is exceedingly narrow-minded.
>> We are talking about partitioning. It is supposed to be narrow-minded.
>
> Sure, but look at all the confusion we have had just on this list about
> it.  We had better state why triggers should be used in place of rules
> _for_ _partitioning_ or that confusion will continue.

Sure I have no problem with that.

Joshua D. Drake

"Joshua D. Drake" <jd@commandprompt.com> writes:

> Tom Lane wrote:
>
>>  A trigger will probably beat a rule for inserts/updates involving a small
>> number of rows.
>
> Which is exactly what partitioning is doing.

Say what?

--
  Gregory Stark
  EnterpriseDB          http://www.enterprisedb.com
  Ask me about EnterpriseDB's On-Demand Production Tuning

"Joshua D. Drake" <jd@commandprompt.com> writes:
> Tom Lane wrote:
>> A trigger
>> will probably beat a rule for inserts/updates involving a small number
>> of rows.

> Which is exactly what partitioning is doing.

Nonsense.  Well, maybe *you* never do that, but if so you are hardly
reflective of the whole world.

            regards, tom lane

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Thu, 29 Nov 2007 17:08:29 +0000
Gregory Stark <stark@enterprisedb.com> wrote:

> 
> "Joshua D. Drake" <jd@commandprompt.com> writes:
> 
> > Tom Lane wrote:
> >
> >>  A trigger will probably beat a rule for inserts/updates involving
> >> a small number of rows.
> >
> > Which is exactly what partitioning is doing.
> 
> Say what?

Heh, o.k. that was an ambiguous sentence. In a partitioned environment
you are likely not moving millions of rows around. Thus the "rule"
benefit is lost. You are instead performing many (sometimes
lots-o-many) inserts and updates that involve a small number of rows.

A trigger/function as Tom already pointed out is going to perform better
than a rule in that case. The benefit becomes even more pronounce the
more partitions you have.

Sincerely,

Joshua D. Drake


- -- 

      === The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564   24x7/Emergency: +1.800.492.2240
PostgreSQL solutions since 1997  http://www.commandprompt.com/
            UNIQUE NOT NULL
Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
PostgreSQL Replication: http://www.commandprompt.com/products/

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFHTvNXATb/zqfZUUQRAqeJAJ9CoO6F4zYJwY4geAR1UsvVCYqagwCdFJm9
63GRksppATp7fK9qu2RYXD0=
=VuWy
-----END PGP SIGNATURE-----

Joshua D. Drake wrote:
>
>>  A trigger
>> will probably beat a rule for inserts/updates involving a small number
>> of rows.
>
> Which is exactly what partitioning is doing.
>
>  For large numbers of rows, like an INSERT/SELECT from another
>> large table, the rule is likely to win, because its overhead is paid
>> once per query not once per row.  Also, if you implement the trigger
>> with an EXECUTE (forcing a planning cycle) intead of hard-coded
>> commands, the speed advantage becomes even more dubious.
>
> Not for partitioning. Although I agree with your sentiments for normal
> operation.
>
>

Joshua, you're not making much sense here.

Tom is talking about partitioning and his analysis is correct *in the
partitioning case* AFAICS.

What basis do you have for saying he is not?

cheers

andrew

"Joshua D. Drake" <jd@commandprompt.com> writes:

> Heh, o.k. that was an ambiguous sentence. In a partitioned environment
> you are likely not moving millions of rows around. Thus the "rule"
> benefit is lost. You are instead performing many (sometimes
> lots-o-many) inserts and updates that involve a small number of rows.

I'm still not following at all. If you're partitioning it's because you have a
*lot* of data. It doesn't say anything about what you're doing with that data.
Partitioning is useful for managing large quantities of data for both OLTP and
DSS systems.

I tend to be happier recommending triggers over rules if only because rules
are just harder to understand. Arguably they don't really work properly for
this use anyways given what happens if you use volatile functions like
random() in your where clause.

--
  Gregory Stark
  EnterpriseDB          http://www.enterprisedb.com
  Ask me about EnterpriseDB's On-Demand Production Tuning

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Thu, 29 Nov 2007 17:29:51 +0000
Gregory Stark <stark@enterprisedb.com> wrote:

> "Joshua D. Drake" <jd@commandprompt.com> writes:
> 
> > Heh, o.k. that was an ambiguous sentence. In a partitioned
> > environment you are likely not moving millions of rows around. Thus
> > the "rule" benefit is lost. You are instead performing many
> > (sometimes lots-o-many) inserts and updates that involve a small
> > number of rows.
> 
> I'm still not following at all. If you're partitioning it's because
> you have a *lot* of data. It doesn't say anything about what you're

If you have lots of data it doesn't mean you are modifying lots of
data. I took perhaps incorrectly what tgl said as modifying lots of
data. E.g; I am doing a large transaction that is going to
insert/update 500 thousand rows.

I don't think anyone here (good lord I hope not) would say that firing
a trigger over 500k rows is fast. Instead you should likely just work the data
outside the partition and then move it directly into the target
partition.

> doing with that data. Partitioning is useful for managing large
> quantities of data for both OLTP and DSS systems.

Certainly. I am not really arguing that and I would tend to agree that
I am being very focused on my arguments about partitioning. To me it is
obvious that you don't use triggers or rules when moving tons of data,
either one is just a burden you don't need.

Partitioning is generally most useful for:

Breaking up large tables and indexes so you are dealing with less data
on active queries.

Breaking up large tables so you don't end up vacuuming at 500 million
row table that only 1 million rows are ever updated.

Rotating out highly updated data so you can keep bloat down (HOT
resolves this in certain cases).

General data management of large sets. Archives and the like.

In any of the above cases a trigger is going to work better than a rule
with the exceptions of what TGL pointed out and in simpler
partitioning environments where the number of partitions are very low.

Either way, to drive this back to topic :).. on the docs if we keep the
rule example it should be below the partitioned example and we need to
list caveats on both. Which I personally think is overkill for
reference documentation but it seems to be what some people want.

> 
> I tend to be happier recommending triggers over rules if only because
> rules are just harder to understand. Arguably they don't really work
> properly for this use anyways given what happens if you use volatile
> functions like random() in your where clause.
> 

nod.

Sincerely,

Joshua D. Drake



- -- 

      === The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564   24x7/Emergency: +1.800.492.2240
PostgreSQL solutions since 1997  http://www.commandprompt.com/
            UNIQUE NOT NULL
Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
PostgreSQL Replication: http://www.commandprompt.com/products/

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFHTvxkATb/zqfZUUQRAhRtAKCHcWBhVZgTM8XXq8kJWWmi0m49cACgmAU5
xqosTo6sJPqpMMKWNvoTWGU=
=zDUw
-----END PGP SIGNATURE-----

On Thu, Nov 29, 2007 at 11:42:18AM -0500, Bruce Momjian wrote:
> Joshua D. Drake wrote:
> > Tom Lane wrote:
> > > "Joshua D. Drake" <jd@commandprompt.com> writes:
> > >> Rules are extremely slow in comparisons and not anywhere near as
> > >> flexible. As I said up post yesterday... they work well in the basic
> > >> partitioning configuration but anything else they are extremely deficient.
> > >
> > > I think that the above claim is exceedingly narrow-minded.
> >
> > We are talking about partitioning. It is supposed to be narrow-minded.
>
> Sure, but look at all the confusion we have had just on this list about
> it.  We had better state why triggers should be used in place of rules
> _for_ _partitioning_ or that confusion will continue.

Please find enclosed a patch with use cases for each.

Cheers,
David.
--
David Fetter <david@fetter.org> http://fetter.org/
Phone: +1 415 235 3778  AIM: dfetter666  Yahoo!: dfetter
Skype: davidfetter      XMPP: david.fetter@gmail.com

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

"Joshua D. Drake" <jd@commandprompt.com> writes:

> If you have lots of data it doesn't mean you are modifying lots of
> data.

It sure can. How do you modify lots of data if you *don't* have lots data in
the first place? Certainly it doesn't mean you necessarily are, some databases
are OLTP which do no large updates. But data warehouses with oodles of data
also often have to do large updates or deletions.

> I don't think anyone here (good lord I hope not) would say that firing
> a trigger over 500k rows is fast. Instead you should likely just work the data
> outside the partition and then move it directly into the target
> partition.

Well you don't even have to do that. You can issue the updates directly
against the partitions. In fact, that's precisely what the rules effectively
do... Rules rewrite the query to be a query directly against the partitions.

Come to think of it I think there actually is a correct way to use rules which
wouldn't suffer from the problems that have come up. Instead of putting a
WHERE clause on the rule just expand deletes and updates to expand to deletes
and updates against *all* partitions. Then let constraint_exclusion kick in to
narrow down which partitions should actually receive the updates and deletes.
I think triggers are the only solution for insert though.

--
  Gregory Stark
  EnterpriseDB          http://www.enterprisedb.com
  Ask me about EnterpriseDB's 24x7 Postgres support!

On Thu, 2007-11-29 at 09:52 -0800, Joshua D. Drake wrote:

> In any of the above cases a trigger is going to work better than a
> rule with the exceptions of what TGL pointed out and in simpler
> partitioning environments where the number of partitions are very low.

Agreed to this and in general to JD's points.


Tom's point about Rules being statement-level is valid only when all
rows from an INSERT SELECT go into one partition. If that were the case
then it seems strange to spend lots of time designing a rules
infrastructure anyway.

If you try to do lots of partitions and RULEs then it sucks. I spoke
against their inclusion originally and do so again now. My point then
was you don't need this for most partitioning applications.

What nobody has mentioned is that Triggers suck as well, so neither
Triggers nor Rules should be given centre stage. COPY only makes sense
running into the table you are loading and if you're trying to load
large amounts of data using INSERTs + anything then you need a whack.
Rob did a beautiful de-construction of all of this in Montreal, BTW,
with humour too.

Current PostgreSQL partitioning is not the same as Oracle's and papering
over the cracks doesn't help anybody much. ISTM we should say to people
to use COPY into a named partition for high speed, plus these other
suggestions if you want some fancy logic, but go careful, cos they're
slow.

It would be nice to have an example of using CREATE TABLE LIKE + COPY in
same transaction, then ALTER TABLE ... INHERITS to add the partition
onto the main table. That is now the fastest way in 8.3.

I'll leave it to y'all from here though.

--
  Simon Riggs
  2ndQuadrant  http://www.2ndQuadrant.com

On Fri, Nov 30, 2007 at 12:34:05PM +0530, NikhilS wrote:
> Hi,
>
> Another reason to go along with triggers is that "COPY" honors
> triggers, but does not honor rules. While trying to do bulk inserts
> into a parent of partitioned tables where rules are being employed,
> the COPY operation will not be so straightforward.

Folks,

Does my latest patch attached address this well enough?

Cheers,
David.
--
David Fetter <david@fetter.org> http://fetter.org/
Phone: +1 415 235 3778  AIM: dfetter666  Yahoo!: dfetter
Skype: davidfetter      XMPP: david.fetter@gmail.com

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

David Fetter <david@fetter.org> writes:
> On Fri, Nov 30, 2007 at 12:34:05PM +0530, NikhilS wrote:
>> Another reason to go along with triggers is that "COPY" honors
>> triggers, but does not honor rules. While trying to do bulk inserts
>> into a parent of partitioned tables where rules are being employed,
>> the COPY operation will not be so straightforward.

> Does my latest patch attached address this well enough?

Applied with revisions and extensions.  (I take it you hadn't actually
tested the example :-()

            regards, tom lane