Thread: CLUSTER vs. VACUUM FULL

CLUSTER vs. VACUUM FULL

From
Ron Johnson
Date:
PG 14.11 on RHEL8

Why is VACUUM FULL recommended for compressing a table, when CLUSTER does the same thing (similarly doubling disk space), and apparently runs just as fast?  

My tests:
Table: CDSLBXW.public.log

            Time 1  Time 2  Time 3
            secs    secs    secs
VACUUM FULL 44.2    39.3    42.3
CLUSTER     41.7    38.9    41.3

CDSLBXW=# \d public.log
                                             Table "public.log"
     Column      |            Type             | Collation | Nullable |               Default              
-----------------+-----------------------------+-----------+----------+-------------------------------------
 log_id          | bigint                      |           | not null | nextval('log_log_id_seq'::regclass)
 level           | numeric(10,0)               |           |          |
 source          | character varying(255)      |           |          |
 username        | character varying(255)      |           |          |
 user_login_id   | character varying(255)      |           |          |
 user_ip_address | character varying(255)      |           |          |
 computer        | character varying(255)      |           |          |
 search_tag      | character varying(4000)     |           |          |
 log_group_id    | integer                     |           |          |
 created_on      | timestamp without time zone |           | not null |
 created_by      | integer                     |           |          |
 xml_detail      | bytea                       |           |          |
Indexes:
    "pk_log" PRIMARY KEY, btree (log_id)
    "idx_log_attr_source" btree (source)
    "idx_log_level" btree (level)
    "idx_log_search_tag" btree (search_tag)

CDSLBXW=# SELECT COUNT(*) FROM public.log;
  count  
---------
 6774664
(1 row)

CDSLBXW=# \timing
CDSLBXW=# vacuum (full) public.log;
VACUUM
Time: 44190.799 ms (00:44.191)
CDSLBXW=#
CDSLBXW=# vacuum (full) public.log;
VACUUM
Time: 39316.824 ms (00:39.317)
CDSLBXW=# vacuum (full) public.log;
VACUUM
Time: 42336.268 ms (00:42.336)

CDSLBXW=# CLUSTER public.log USING pk_log;
CLUSTER
Time: 41722.335 ms (00:41.722)
CDSLBXW=#
CDSLBXW=# CLUSTER public.log USING pk_log;
CLUSTER
Time: 38915.128 ms (00:38.915)
CDSLBXW=#
CDSLBXW=# CLUSTER public.log USING pk_log;
CLUSTER
Time: 41342.651 ms (00:41.343)


Re: CLUSTER vs. VACUUM FULL

From
Tom Lane
Date:
Ron Johnson <ronljohnsonjr@gmail.com> writes:
> Why is VACUUM FULL recommended for compressing a table, when CLUSTER does
> the same thing (similarly doubling disk space), and apparently runs just as
> fast?

CLUSTER makes the additional effort to sort the data per the ordering
of the specified index.  I'm surprised that's not noticeable in your
test case.

            regards, tom lane



Re: CLUSTER vs. VACUUM FULL

From
Ron Johnson
Date:
On Sun, Apr 21, 2024 at 6:45 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
Ron Johnson <ronljohnsonjr@gmail.com> writes:
> Why is VACUUM FULL recommended for compressing a table, when CLUSTER does
> the same thing (similarly doubling disk space), and apparently runs just as
> fast?

CLUSTER makes the additional effort to sort the data per the ordering
of the specified index.  I'm surprised that's not noticeable in your
test case.

It's in a freshly restored database.  However, regular deletions of old records, and normal vacuums would have led me to expect that the "fresh" public.log would have been in relatively random order.

Re: CLUSTER vs. VACUUM FULL

From
Ron Johnson
Date:
On Sun, Apr 21, 2024 at 6:45 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
Ron Johnson <ronljohnsonjr@gmail.com> writes:
> Why is VACUUM FULL recommended for compressing a table, when CLUSTER does
> the same thing (similarly doubling disk space), and apparently runs just as
> fast?

CLUSTER makes the additional effort to sort the data per the ordering
of the specified index.  I'm surprised that's not noticeable in your
test case.

Clustering on a completely different index  was also 44 seconds.

Re: CLUSTER vs. VACUUM FULL

From
David Rowley
Date:
On Mon, 22 Apr 2024 at 12:16, Ron Johnson <ronljohnsonjr@gmail.com> wrote:
>
> On Sun, Apr 21, 2024 at 6:45 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>>
>> Ron Johnson <ronljohnsonjr@gmail.com> writes:
>> > Why is VACUUM FULL recommended for compressing a table, when CLUSTER does
>> > the same thing (similarly doubling disk space), and apparently runs just as
>> > fast?
>>
>> CLUSTER makes the additional effort to sort the data per the ordering
>> of the specified index.  I'm surprised that's not noticeable in your
>> test case.
>
> Clustering on a completely different index  was also 44 seconds.

Both VACUUM FULL and CLUSTER go through a very similar code path. Both
use cluster_rel().  VACUUM FULL just won't make use of an existing
index to provide presorted input or perform a sort, whereas CLUSTER
will attempt to choose the cheapest out of these two to get sorted
results.

If the timing for each is similar, it just means that using an index
scan or sorting isn't very expensive compared to the other work that's
being done.  Both CLUSTER and VACUUM FULL require reading every heap
page and writing out new pages into a new heap and maintaining  all
indexes on the new heap. That's quite an effort.

To satisfy your curiosity, you could always run some EXPLAIN ANALYZE
SELECT queries to measure how much time was spent sorting the entire
table. You'd have to set work_mem to the value of
maintenance_work_mem.

David



Re: CLUSTER vs. VACUUM FULL

From
Ron Johnson
Date:
On Sun, Apr 21, 2024 at 9:35 PM David Rowley <dgrowleyml@gmail.com> wrote:
On Mon, 22 Apr 2024 at 12:16, Ron Johnson <ronljohnsonjr@gmail.com> wrote:
>
> On Sun, Apr 21, 2024 at 6:45 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>>
>> Ron Johnson <ronljohnsonjr@gmail.com> writes:
>> > Why is VACUUM FULL recommended for compressing a table, when CLUSTER does
>> > the same thing (similarly doubling disk space), and apparently runs just as
>> > fast?
>>
>> CLUSTER makes the additional effort to sort the data per the ordering
>> of the specified index.  I'm surprised that's not noticeable in your
>> test case.
>
> Clustering on a completely different index  was also 44 seconds.

Both VACUUM FULL and CLUSTER go through a very similar code path. Both
use cluster_rel().  VACUUM FULL just won't make use of an existing
index to provide presorted input or perform a sort, whereas CLUSTER
will attempt to choose the cheapest out of these two to get sorted
results.

If the timing for each is similar, it just means that using an index
scan or sorting isn't very expensive compared to the other work that's
being done.  Both CLUSTER and VACUUM FULL require reading every heap
page and writing out new pages into a new heap and maintaining  all
indexes on the new heap. That's quite an effort.

My original CLUSTER command didn't have to change the order of the data very much, thus, the sort didn't have to do much work.

CLUSTER on a different index was indeed much slower than VACUUM FULL.

Re: CLUSTER vs. VACUUM FULL

From
Marcos Pegoraro
Date:
Em dom., 21 de abr. de 2024 às 22:35, David Rowley <dgrowleyml@gmail.com> escreveu:

Both VACUUM FULL and CLUSTER go through a very similar code path. Both
use cluster_rel().  VACUUM FULL just won't make use of an existing
index to provide presorted input or perform a sort, whereas CLUSTER
will attempt to choose the cheapest out of these two to get sorted
results.

But wouldn't it be good that VACUUM FULL uses that index defined by Cluster, if it exists ? Maybe an additional option for VACUUM FULL ?
Because if I periodically reorganize all tables I have to run CLUSTER once, which will run on clustered tables, and VACUUM FULL on every table that is not clustered, because if I run VACUUM FULL for entire database it'll just ignore cluster index defined for every table. So, INDISCLUSTERED is used when running CLUSTER but is ignored when running VACUUM FULL.

regards
Marcos

Re: CLUSTER vs. VACUUM FULL

From
Tom Lane
Date:
Marcos Pegoraro <marcos@f10.com.br> writes:
> But wouldn't it be good that VACUUM FULL uses that index defined by
> Cluster, if it exists ?

No ... what would be the difference then?

            regards, tom lane



Re: CLUSTER vs. VACUUM FULL

From
Ron Johnson
Date:
On Mon, Apr 22, 2024 at 10:25 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
Marcos Pegoraro <marcos@f10.com.br> writes:
> But wouldn't it be good that VACUUM FULL uses that index defined by
> Cluster, if it exists ?

No ... what would be the difference then?
 
What the VACUUM docs "should" do, it seems, is suggest CLUSTER on the PK, if the PK is a sequence (whether that be an actual sequence, or a timestamp or something else that grows monotonically).

That's because the data is already roughly in PK order.

Re: CLUSTER vs. VACUUM FULL

From
Adrian Klaver
Date:
On 4/22/24 08:37, Ron Johnson wrote:
> On Mon, Apr 22, 2024 at 10:25 AM Tom Lane <tgl@sss.pgh.pa.us 
> <mailto:tgl@sss.pgh.pa.us>> wrote:
> 
>     Marcos Pegoraro <marcos@f10.com.br <mailto:marcos@f10.com.br>> writes:
>      > But wouldn't it be good that VACUUM FULL uses that index defined by
>      > Cluster, if it exists ?
> 
>     No ... what would be the difference then?
> 
> What the VACUUM docs "should" do, it seems, is suggest CLUSTER on the 
> PK, if the PK is a sequence (whether that be an actual sequence, or a 
> timestamp or something else that grows monotonically).

Why?

That would, per David Rowley's comments, impose a sort cost on top of 
the cost of hitting every heap page and rewriting it. You end up with 
sorted table granted, until such time as you start making changes to it. 
If you are to the point of running VACUUM FULL that indicates to me the 
table has seen a heavy load of changes that you want to clean out. Given 
the temporary nature of the effects of a  CLUSTER under a change load I 
don't see why it would be the way to go to clean up a changing table.

> 
> That's because the data is already roughly in PK order.
> 

-- 
Adrian Klaver
adrian.klaver@aklaver.com




Re: CLUSTER vs. VACUUM FULL

From
"David G. Johnston"
Date:


On Mon, Apr 22, 2024, 08:37 Ron Johnson <ronljohnsonjr@gmail.com> wrote:
On Mon, Apr 22, 2024 at 10:25 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
Marcos Pegoraro <marcos@f10.com.br> writes:
> But wouldn't it be good that VACUUM FULL uses that index defined by
> Cluster, if it exists ?

No ... what would be the difference then?
 
What the VACUUM docs "should" do, it seems, is suggest CLUSTER on the PK, if the PK is a sequence (whether that be an actual sequence, or a timestamp or something else that grows monotonically).

That's because the data is already roughly in PK order.

If things are bad enough to require a vacuum full that doesn't seem like a good assumption.  Any insert-only table or one with a reduced fill-factor maybe.

David J 

Re: CLUSTER vs. VACUUM FULL

From
Marcos Pegoraro
Date:
Em seg., 22 de abr. de 2024 às 11:25, Tom Lane <tgl@sss.pgh.pa.us> escreveu:
No ... what would be the difference then

Well, I think if a cluster index was defined sometime in a table, it should be respected for next commands, including VACUUM FULL. 
If I want to come back to PK or any other index I would use CLUSTER ... USING PK_INDEX.

regards
Marcos

Re: CLUSTER vs. VACUUM FULL

From
Ron Johnson
Date:
On Mon, Apr 22, 2024 at 12:29 PM David G. Johnston <david.g.johnston@gmail.com> wrote:


On Mon, Apr 22, 2024, 08:37 Ron Johnson <ronljohnsonjr@gmail.com> wrote:
On Mon, Apr 22, 2024 at 10:25 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
Marcos Pegoraro <marcos@f10.com.br> writes:
> But wouldn't it be good that VACUUM FULL uses that index defined by
> Cluster, if it exists ?

No ... what would be the difference then?
 
What the VACUUM docs "should" do, it seems, is suggest CLUSTER on the PK, if the PK is a sequence (whether that be an actual sequence, or a timestamp or something else that grows monotonically).

That's because the data is already roughly in PK order.

If things are bad enough to require a vacuum full that doesn't seem like a good assumption.

Sure it does.

For example, I just deleted the oldest half of the records in 30 tables.  Tables who's CREATED_ON timestamp value strongly correlates to the synthetic PK sequence values. 

Thus, the remaining records were still mostly in PK order.  CLUSTERs on the PK values would have taken just about as much time as the VACUUM FULL statements which I did run.

Re: CLUSTER vs. VACUUM FULL

From
Adrian Klaver
Date:

On 4/22/24 11:45 AM, Ron Johnson wrote:
> On Mon, Apr 22, 2024 at 12:29 PM David G. Johnston 
> <david.g.johnston@gmail.com <mailto:david.g.johnston@gmail.com>> wrote:
> 
> 
> 
>     On Mon, Apr 22, 2024, 08:37 Ron Johnson <ronljohnsonjr@gmail.com
>     <mailto:ronljohnsonjr@gmail.com>> wrote:
> 
>         On Mon, Apr 22, 2024 at 10:25 AM Tom Lane <tgl@sss.pgh.pa.us
>         <mailto:tgl@sss.pgh.pa.us>> wrote:
> 
>             Marcos Pegoraro <marcos@f10.com.br
>             <mailto:marcos@f10.com.br>> writes:
>              > But wouldn't it be good that VACUUM FULL uses that index
>             defined by
>              > Cluster, if it exists ?
> 
>             No ... what would be the difference then?
> 
>         What the VACUUM docs "should" do, it seems, is suggest CLUSTER
>         on the PK, if the PK is a sequence (whether that be an actual
>         sequence, or a timestamp or something else that grows
>         monotonically).
> 
>         That's because the data is already roughly in PK order.
> 
> 
>     If things are bad enough to require a vacuum full that doesn't seem
>     like a good assumption.
> 
> 
> Sure it does.
> 
> For example, I just deleted the oldest half of the records in 30 
> tables.  Tables who's CREATED_ON timestamp value strongly correlates to 
> the synthetic PK sequence values.
> 
> Thus, the remaining records were still mostly in PK order.  CLUSTERs on 
> the PK values would have taken just about as much time as the VACUUM 
> FULL statements which I /did/ run.

1) If they are already in enough of a PK order that the CLUSTER time vs 
VACUUM FULL time would not be material as there is not much or any 
sorting to do then what does the CLUSTER gain you? Unless this table 
then became read only whatever small gain arose from the CLUSTER would 
fade away as UPDATEs and DELETEs where done.

2) What evidence is there that the records where still in PK order just 
because you deleted based on CREATED_ON? I understand the correlation 
between CREATED_ON and the PK just not sure why that would necessarily 
translate to an on disk order by PK?

-- 
Adrian Klaver
adrian.klaver@aklaver.com



Re: CLUSTER vs. VACUUM FULL

From
Ron Johnson
Date:
On Mon, Apr 22, 2024 at 3:14 PM Adrian Klaver <adrian.klaver@aklaver.com> wrote:


On 4/22/24 11:45 AM, Ron Johnson wrote:
> On Mon, Apr 22, 2024 at 12:29 PM David G. Johnston
> <david.g.johnston@gmail.com <mailto:david.g.johnston@gmail.com>> wrote:
>
>
>
>     On Mon, Apr 22, 2024, 08:37 Ron Johnson <ronljohnsonjr@gmail.com
>     <mailto:ronljohnsonjr@gmail.com>> wrote:
>
>         On Mon, Apr 22, 2024 at 10:25 AM Tom Lane <tgl@sss.pgh.pa.us
>         <mailto:tgl@sss.pgh.pa.us>> wrote:
>
>             Marcos Pegoraro <marcos@f10.com.br
>             <mailto:marcos@f10.com.br>> writes:
>              > But wouldn't it be good that VACUUM FULL uses that index
>             defined by
>              > Cluster, if it exists ?
>
>             No ... what would be the difference then?
>
>         What the VACUUM docs "should" do, it seems, is suggest CLUSTER
>         on the PK, if the PK is a sequence (whether that be an actual
>         sequence, or a timestamp or something else that grows
>         monotonically).
>
>         That's because the data is already roughly in PK order.
>
>
>     If things are bad enough to require a vacuum full that doesn't seem
>     like a good assumption.
>
>
> Sure it does.
>
> For example, I just deleted the oldest half of the records in 30
> tables.  Tables who's CREATED_ON timestamp value strongly correlates to
> the synthetic PK sequence values.
>
> Thus, the remaining records were still mostly in PK order.  CLUSTERs on
> the PK values would have taken just about as much time as the VACUUM
> FULL statements which I /did/ run.

1) If they are already in enough of a PK order that the CLUSTER time vs
VACUUM FULL time would not be material as there is not much or any
sorting to do then what does the CLUSTER gain you?

Not much.  Now they're just "slightly more ordered" instead of "slightly less ordered" for little if any extra effort.
 
2) What evidence is there that the records where still in PK order just
because you deleted based on CREATED_ON? I understand the correlation
between CREATED_ON and the PK just not sure why that would necessarily
translate to an on disk order by PK?

1. Records are appended to tables in INSERT order, and INSERT order is highly correlated to synthetic PK, by the nature of sequences.
2. My original email showed that CLUSTER took just as long as VACUUM FULL.  That means not many records had to be sorted, because... the on-disk order was strongly correlated to PK and CREATED_ON.

Will that happen every time in every circumstance in every database?  No, and I never said it would.  But it does in my database in this application.

Re: CLUSTER vs. VACUUM FULL

From
Adrian Klaver
Date:
On 4/22/24 12:51, Ron Johnson wrote:
> On Mon, Apr 22, 2024 at 3:14 PM Adrian Klaver <adrian.klaver@aklaver.com 
> <mailto:adrian.klaver@aklaver.com>> wrote:
> 
> 
> 

> 
>     1) If they are already in enough of a PK order that the CLUSTER time vs
>     VACUUM FULL time would not be material as there is not much or any
>     sorting to do then what does the CLUSTER gain you? 
> 
> 
> Not much.  Now they're just "slightly more ordered" instead of "slightly 
> less ordered" for little if any extra effort.
> 
>     2) What evidence is there that the records where still in PK order just
>     because you deleted based on CREATED_ON? I understand the correlation
>     between CREATED_ON and the PK just not sure why that would necessarily
>     translate to an on disk order by PK?
> 
> 
> 1. Records are appended to tables in INSERT order, and INSERT order is 
> highly correlated to synthetic PK, by the nature of sequences.

Not something I would count on, see:

https://www.postgresql.org/docs/current/sql-createsequence.html

Notes

for how that may not always be the case.

Also any UPDATE or DELETE is going to change that. There is no guarantee 
of order for the data in the table. If there where you would not need to 
run CLUSTER.

> 2. My original email showed that CLUSTER took just as long as VACUUM 
> FULL.  That means not many records had to be sorted, because... the 
> on-disk order was strongly correlated to PK and CREATED_ON. >
> Will that happen *every time* in *every circumstance* in *every 
> database*?  No, and I never said it would.  But it does in *my *database 
> in *this *application.
> 

Which gets us back to your comment upstream:

"What the VACUUM docs "should" do, it seems, is suggest CLUSTER on the 
PK, if the PK is a sequence (whether that be an actual sequence, or a 
timestamp or something else that grows monotonically)."

This is a case specific to you and this particular circumstance, not a 
general rule for VACUUM FULL. If for no other reason then it might make 
more sense for the application that the CLUSTER be done on some other 
index then the PK.



-- 
Adrian Klaver
adrian.klaver@aklaver.com




Re: CLUSTER vs. VACUUM FULL

From
Ron Johnson
Date:
On Mon, Apr 22, 2024 at 4:21 PM Adrian Klaver <adrian.klaver@aklaver.com> wrote:
[snip] 
Which gets us back to your comment upstream:

"What the VACUUM docs "should" do, it seems, is suggest CLUSTER on the
PK, if the PK is a sequence (whether that be an actual sequence, or a
timestamp or something else that grows monotonically)."

This is a case specific to you and this particular circumstance, not a
general rule for VACUUM FULL. If for no other reason then it might make
more sense for the application that the CLUSTER be done on some other
index then the PK.

On Stack Exchange, I've got a question on how to determine when to run CLUSTER.  It ties in strongly with this thread..

Re: CLUSTER vs. VACUUM FULL

From
Adrian Klaver
Date:
On 4/22/24 13:59, Ron Johnson wrote:
> On Mon, Apr 22, 2024 at 4:21 PM Adrian Klaver <adrian.klaver@aklaver.com 
> <mailto:adrian.klaver@aklaver.com>> wrote:
> [snip]
> 
>     Which gets us back to your comment upstream:
> 
>     "What the VACUUM docs "should" do, it seems, is suggest CLUSTER on the
>     PK, if the PK is a sequence (whether that be an actual sequence, or a
>     timestamp or something else that grows monotonically)."
> 
>     This is a case specific to you and this particular circumstance, not a
>     general rule for VACUUM FULL. If for no other reason then it might make
>     more sense for the application that the CLUSTER be done on some other
>     index then the PK.
> 
> 
> On Stack Exchange, I've got a question on how to determine when to run 
> CLUSTER.  It ties in strongly with this thread..
> 

And the link is?

-- 
Adrian Klaver
adrian.klaver@aklaver.com




Re: CLUSTER vs. VACUUM FULL

From
Ron Johnson
Date:
On Mon, Apr 22, 2024 at 5:03 PM Adrian Klaver <adrian.klaver@aklaver.com> wrote:
On 4/22/24 13:59, Ron Johnson wrote:
> On Mon, Apr 22, 2024 at 4:21 PM Adrian Klaver <adrian.klaver@aklaver.com
> <mailto:adrian.klaver@aklaver.com>> wrote:
> [snip]
>
>     Which gets us back to your comment upstream:
>
>     "What the VACUUM docs "should" do, it seems, is suggest CLUSTER on the
>     PK, if the PK is a sequence (whether that be an actual sequence, or a
>     timestamp or something else that grows monotonically)."
>
>     This is a case specific to you and this particular circumstance, not a
>     general rule for VACUUM FULL. If for no other reason then it might make
>     more sense for the application that the CLUSTER be done on some other
>     index then the PK.
>
>
> On Stack Exchange, I've got a question on how to determine when to run
> CLUSTER.  It ties in strongly with this thread..
>

And the link is?

Sorry.  Got distracted by the answer.

 
 

Re: CLUSTER vs. VACUUM FULL

From
Adrian Klaver
Date:
On 4/22/24 14:35, Ron Johnson wrote:

>      >
>      > On Stack Exchange, I've got a question on how to determine when
>     to run
>      > CLUSTER.  It ties in strongly with this thread..
>      >
> 
>     And the link is?

It should have been the initial question of this thread and it explains 
what you are really after.

> 
> 
> Sorry.  Got distracted by the answer.
> 
> https://dba.stackexchange.com/questions/338870/when-to-rerun-cluster 
> <https://dba.stackexchange.com/questions/338870/when-to-rerun-cluster>



-- 
Adrian Klaver
adrian.klaver@aklaver.com




Re: CLUSTER vs. VACUUM FULL

From
Ron Johnson
Date:
On Mon, Apr 22, 2024 at 5:56 PM Adrian Klaver <adrian.klaver@aklaver.com> wrote:
On 4/22/24 14:35, Ron Johnson wrote:

>      >
>      > On Stack Exchange, I've got a question on how to determine when
>     to run
>      > CLUSTER.  It ties in strongly with this thread..
>      >
>
>     And the link is?

It should have been the initial question of this thread and it explains
what you are really after.

It was already a long email.