Thread: CLUSTER vs. VACUUM FULL
PG 14.11 on RHEL8
Why is VACUUM FULL recommended for compressing a table, when CLUSTER does the same thing (similarly doubling disk space), and apparently runs just as fast?
My tests:
Table: CDSLBXW.public.log
Time 1 Time 2 Time 3
secs secs secs
VACUUM FULL 44.2 39.3 42.3
CLUSTER 41.7 38.9 41.3
CDSLBXW=# \d public.log
Table "public.log"
Column | Type | Collation | Nullable | Default
-----------------+-----------------------------+-----------+----------+-------------------------------------
log_id | bigint | | not null | nextval('log_log_id_seq'::regclass)
level | numeric(10,0) | | |
source | character varying(255) | | |
username | character varying(255) | | |
user_login_id | character varying(255) | | |
user_ip_address | character varying(255) | | |
computer | character varying(255) | | |
search_tag | character varying(4000) | | |
log_group_id | integer | | |
created_on | timestamp without time zone | | not null |
created_by | integer | | |
xml_detail | bytea | | |
Indexes:
"pk_log" PRIMARY KEY, btree (log_id)
"idx_log_attr_source" btree (source)
"idx_log_level" btree (level)
"idx_log_search_tag" btree (search_tag)
CDSLBXW=# SELECT COUNT(*) FROM public.log;
count
---------
6774664
(1 row)
CDSLBXW=# \timing
Time 1 Time 2 Time 3
secs secs secs
VACUUM FULL 44.2 39.3 42.3
CLUSTER 41.7 38.9 41.3
CDSLBXW=# \d public.log
Table "public.log"
Column | Type | Collation | Nullable | Default
-----------------+-----------------------------+-----------+----------+-------------------------------------
log_id | bigint | | not null | nextval('log_log_id_seq'::regclass)
level | numeric(10,0) | | |
source | character varying(255) | | |
username | character varying(255) | | |
user_login_id | character varying(255) | | |
user_ip_address | character varying(255) | | |
computer | character varying(255) | | |
search_tag | character varying(4000) | | |
log_group_id | integer | | |
created_on | timestamp without time zone | | not null |
created_by | integer | | |
xml_detail | bytea | | |
Indexes:
"pk_log" PRIMARY KEY, btree (log_id)
"idx_log_attr_source" btree (source)
"idx_log_level" btree (level)
"idx_log_search_tag" btree (search_tag)
CDSLBXW=# SELECT COUNT(*) FROM public.log;
count
---------
6774664
(1 row)
CDSLBXW=# \timing
CDSLBXW=# vacuum (full) public.log;
VACUUM
Time: 44190.799 ms (00:44.191)
CDSLBXW=#
CDSLBXW=# vacuum (full) public.log;
VACUUM
Time: 39316.824 ms (00:39.317)
CDSLBXW=# vacuum (full) public.log;
VACUUM
Time: 42336.268 ms (00:42.336)
CDSLBXW=# CLUSTER public.log USING pk_log;
CLUSTER
Time: 41722.335 ms (00:41.722)
CDSLBXW=#
CDSLBXW=# CLUSTER public.log USING pk_log;
CLUSTER
Time: 38915.128 ms (00:38.915)
CDSLBXW=#
CDSLBXW=# CLUSTER public.log USING pk_log;
CLUSTER
Time: 41342.651 ms (00:41.343)
VACUUM
Time: 44190.799 ms (00:44.191)
CDSLBXW=#
CDSLBXW=# vacuum (full) public.log;
VACUUM
Time: 39316.824 ms (00:39.317)
CDSLBXW=# vacuum (full) public.log;
VACUUM
Time: 42336.268 ms (00:42.336)
CDSLBXW=# CLUSTER public.log USING pk_log;
CLUSTER
Time: 41722.335 ms (00:41.722)
CDSLBXW=#
CDSLBXW=# CLUSTER public.log USING pk_log;
CLUSTER
Time: 38915.128 ms (00:38.915)
CDSLBXW=#
CDSLBXW=# CLUSTER public.log USING pk_log;
CLUSTER
Time: 41342.651 ms (00:41.343)
Ron Johnson <ronljohnsonjr@gmail.com> writes:
> Why is VACUUM FULL recommended for compressing a table, when CLUSTER does
> the same thing (similarly doubling disk space), and apparently runs just as
> fast?
CLUSTER makes the additional effort to sort the data per the ordering
of the specified index. I'm surprised that's not noticeable in your
test case.
regards, tom lane
On Sun, Apr 21, 2024 at 6:45 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
Ron Johnson <ronljohnsonjr@gmail.com> writes:
> Why is VACUUM FULL recommended for compressing a table, when CLUSTER does
> the same thing (similarly doubling disk space), and apparently runs just as
> fast?
CLUSTER makes the additional effort to sort the data per the ordering
of the specified index. I'm surprised that's not noticeable in your
test case.
It's in a freshly restored database. However, regular deletions of old records, and normal vacuums would have led me to expect that the "fresh" public.log would have been in relatively random order.
On Sun, Apr 21, 2024 at 6:45 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
Ron Johnson <ronljohnsonjr@gmail.com> writes:
> Why is VACUUM FULL recommended for compressing a table, when CLUSTER does
> the same thing (similarly doubling disk space), and apparently runs just as
> fast?
CLUSTER makes the additional effort to sort the data per the ordering
of the specified index. I'm surprised that's not noticeable in your
test case.
Clustering on a completely different index was also 44 seconds.
On Mon, 22 Apr 2024 at 12:16, Ron Johnson <ronljohnsonjr@gmail.com> wrote: > > On Sun, Apr 21, 2024 at 6:45 PM Tom Lane <tgl@sss.pgh.pa.us> wrote: >> >> Ron Johnson <ronljohnsonjr@gmail.com> writes: >> > Why is VACUUM FULL recommended for compressing a table, when CLUSTER does >> > the same thing (similarly doubling disk space), and apparently runs just as >> > fast? >> >> CLUSTER makes the additional effort to sort the data per the ordering >> of the specified index. I'm surprised that's not noticeable in your >> test case. > > Clustering on a completely different index was also 44 seconds. Both VACUUM FULL and CLUSTER go through a very similar code path. Both use cluster_rel(). VACUUM FULL just won't make use of an existing index to provide presorted input or perform a sort, whereas CLUSTER will attempt to choose the cheapest out of these two to get sorted results. If the timing for each is similar, it just means that using an index scan or sorting isn't very expensive compared to the other work that's being done. Both CLUSTER and VACUUM FULL require reading every heap page and writing out new pages into a new heap and maintaining all indexes on the new heap. That's quite an effort. To satisfy your curiosity, you could always run some EXPLAIN ANALYZE SELECT queries to measure how much time was spent sorting the entire table. You'd have to set work_mem to the value of maintenance_work_mem. David
On Sun, Apr 21, 2024 at 9:35 PM David Rowley <dgrowleyml@gmail.com> wrote:
On Mon, 22 Apr 2024 at 12:16, Ron Johnson <ronljohnsonjr@gmail.com> wrote:
>
> On Sun, Apr 21, 2024 at 6:45 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>>
>> Ron Johnson <ronljohnsonjr@gmail.com> writes:
>> > Why is VACUUM FULL recommended for compressing a table, when CLUSTER does
>> > the same thing (similarly doubling disk space), and apparently runs just as
>> > fast?
>>
>> CLUSTER makes the additional effort to sort the data per the ordering
>> of the specified index. I'm surprised that's not noticeable in your
>> test case.
>
> Clustering on a completely different index was also 44 seconds.
Both VACUUM FULL and CLUSTER go through a very similar code path. Both
use cluster_rel(). VACUUM FULL just won't make use of an existing
index to provide presorted input or perform a sort, whereas CLUSTER
will attempt to choose the cheapest out of these two to get sorted
results.
If the timing for each is similar, it just means that using an index
scan or sorting isn't very expensive compared to the other work that's
being done. Both CLUSTER and VACUUM FULL require reading every heap
page and writing out new pages into a new heap and maintaining all
indexes on the new heap. That's quite an effort.
My original CLUSTER command didn't have to change the order of the data very much, thus, the sort didn't have to do much work.
CLUSTER on a different index was indeed much slower than VACUUM FULL.
Em dom., 21 de abr. de 2024 às 22:35, David Rowley <dgrowleyml@gmail.com> escreveu:
Both VACUUM FULL and CLUSTER go through a very similar code path. Both
use cluster_rel(). VACUUM FULL just won't make use of an existing
index to provide presorted input or perform a sort, whereas CLUSTER
will attempt to choose the cheapest out of these two to get sorted
results.
But wouldn't it be good that VACUUM FULL uses that index defined by Cluster, if it exists ? Maybe an additional option for VACUUM FULL ?
Because if I periodically reorganize all tables I have to run CLUSTER once, which will run on clustered tables, and VACUUM FULL on every table that is not clustered, because if I run VACUUM FULL for entire database it'll just ignore cluster index defined for every table. So, INDISCLUSTERED is used when running CLUSTER but is ignored when running VACUUM FULL.
regards
Marcos
Marcos Pegoraro <marcos@f10.com.br> writes:
> But wouldn't it be good that VACUUM FULL uses that index defined by
> Cluster, if it exists ?
No ... what would be the difference then?
regards, tom lane
On Mon, Apr 22, 2024 at 10:25 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
Marcos Pegoraro <marcos@f10.com.br> writes:
> But wouldn't it be good that VACUUM FULL uses that index defined by
> Cluster, if it exists ?
No ... what would be the difference then?
What the VACUUM docs "should" do, it seems, is suggest CLUSTER on the PK, if the PK is a sequence (whether that be an actual sequence, or a timestamp or something else that grows monotonically).
That's because the data is already roughly in PK order.
On 4/22/24 08:37, Ron Johnson wrote: > On Mon, Apr 22, 2024 at 10:25 AM Tom Lane <tgl@sss.pgh.pa.us > <mailto:tgl@sss.pgh.pa.us>> wrote: > > Marcos Pegoraro <marcos@f10.com.br <mailto:marcos@f10.com.br>> writes: > > But wouldn't it be good that VACUUM FULL uses that index defined by > > Cluster, if it exists ? > > No ... what would be the difference then? > > What the VACUUM docs "should" do, it seems, is suggest CLUSTER on the > PK, if the PK is a sequence (whether that be an actual sequence, or a > timestamp or something else that grows monotonically). Why? That would, per David Rowley's comments, impose a sort cost on top of the cost of hitting every heap page and rewriting it. You end up with sorted table granted, until such time as you start making changes to it. If you are to the point of running VACUUM FULL that indicates to me the table has seen a heavy load of changes that you want to clean out. Given the temporary nature of the effects of a CLUSTER under a change load I don't see why it would be the way to go to clean up a changing table. > > That's because the data is already roughly in PK order. > -- Adrian Klaver adrian.klaver@aklaver.com
On Mon, Apr 22, 2024, 08:37 Ron Johnson <ronljohnsonjr@gmail.com> wrote:
On Mon, Apr 22, 2024 at 10:25 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:Marcos Pegoraro <marcos@f10.com.br> writes:
> But wouldn't it be good that VACUUM FULL uses that index defined by
> Cluster, if it exists ?
No ... what would be the difference then?What the VACUUM docs "should" do, it seems, is suggest CLUSTER on the PK, if the PK is a sequence (whether that be an actual sequence, or a timestamp or something else that grows monotonically).That's because the data is already roughly in PK order.
If things are bad enough to require a vacuum full that doesn't seem like a good assumption. Any insert-only table or one with a reduced fill-factor maybe.
David J
Em seg., 22 de abr. de 2024 às 11:25, Tom Lane <tgl@sss.pgh.pa.us> escreveu:
No ... what would be the difference then
Well, I think if a cluster index was defined sometime in a table, it should be respected for next commands, including VACUUM FULL.
If I want to come back to PK or any other index I would use CLUSTER ... USING PK_INDEX.
regards
Marcos
On Mon, Apr 22, 2024 at 12:29 PM David G. Johnston <david.g.johnston@gmail.com> wrote:
On Mon, Apr 22, 2024, 08:37 Ron Johnson <ronljohnsonjr@gmail.com> wrote:On Mon, Apr 22, 2024 at 10:25 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:Marcos Pegoraro <marcos@f10.com.br> writes:
> But wouldn't it be good that VACUUM FULL uses that index defined by
> Cluster, if it exists ?
No ... what would be the difference then?What the VACUUM docs "should" do, it seems, is suggest CLUSTER on the PK, if the PK is a sequence (whether that be an actual sequence, or a timestamp or something else that grows monotonically).That's because the data is already roughly in PK order.If things are bad enough to require a vacuum full that doesn't seem like a good assumption.
Sure it does.
For example, I just deleted the oldest half of the records in 30 tables. Tables who's CREATED_ON timestamp value strongly correlates to the synthetic PK sequence values.
Thus, the remaining records were still mostly in PK order. CLUSTERs on the PK values would have taken just about as much time as the VACUUM FULL statements which I did run.
On 4/22/24 11:45 AM, Ron Johnson wrote: > On Mon, Apr 22, 2024 at 12:29 PM David G. Johnston > <david.g.johnston@gmail.com <mailto:david.g.johnston@gmail.com>> wrote: > > > > On Mon, Apr 22, 2024, 08:37 Ron Johnson <ronljohnsonjr@gmail.com > <mailto:ronljohnsonjr@gmail.com>> wrote: > > On Mon, Apr 22, 2024 at 10:25 AM Tom Lane <tgl@sss.pgh.pa.us > <mailto:tgl@sss.pgh.pa.us>> wrote: > > Marcos Pegoraro <marcos@f10.com.br > <mailto:marcos@f10.com.br>> writes: > > But wouldn't it be good that VACUUM FULL uses that index > defined by > > Cluster, if it exists ? > > No ... what would be the difference then? > > What the VACUUM docs "should" do, it seems, is suggest CLUSTER > on the PK, if the PK is a sequence (whether that be an actual > sequence, or a timestamp or something else that grows > monotonically). > > That's because the data is already roughly in PK order. > > > If things are bad enough to require a vacuum full that doesn't seem > like a good assumption. > > > Sure it does. > > For example, I just deleted the oldest half of the records in 30 > tables. Tables who's CREATED_ON timestamp value strongly correlates to > the synthetic PK sequence values. > > Thus, the remaining records were still mostly in PK order. CLUSTERs on > the PK values would have taken just about as much time as the VACUUM > FULL statements which I /did/ run. 1) If they are already in enough of a PK order that the CLUSTER time vs VACUUM FULL time would not be material as there is not much or any sorting to do then what does the CLUSTER gain you? Unless this table then became read only whatever small gain arose from the CLUSTER would fade away as UPDATEs and DELETEs where done. 2) What evidence is there that the records where still in PK order just because you deleted based on CREATED_ON? I understand the correlation between CREATED_ON and the PK just not sure why that would necessarily translate to an on disk order by PK? -- Adrian Klaver adrian.klaver@aklaver.com
On Mon, Apr 22, 2024 at 3:14 PM Adrian Klaver <adrian.klaver@aklaver.com> wrote:
On 4/22/24 11:45 AM, Ron Johnson wrote:
> On Mon, Apr 22, 2024 at 12:29 PM David G. Johnston
> <david.g.johnston@gmail.com <mailto:david.g.johnston@gmail.com>> wrote:
>
>
>
> On Mon, Apr 22, 2024, 08:37 Ron Johnson <ronljohnsonjr@gmail.com
> <mailto:ronljohnsonjr@gmail.com>> wrote:
>
> On Mon, Apr 22, 2024 at 10:25 AM Tom Lane <tgl@sss.pgh.pa.us
> <mailto:tgl@sss.pgh.pa.us>> wrote:
>
> Marcos Pegoraro <marcos@f10.com.br
> <mailto:marcos@f10.com.br>> writes:
> > But wouldn't it be good that VACUUM FULL uses that index
> defined by
> > Cluster, if it exists ?
>
> No ... what would be the difference then?
>
> What the VACUUM docs "should" do, it seems, is suggest CLUSTER
> on the PK, if the PK is a sequence (whether that be an actual
> sequence, or a timestamp or something else that grows
> monotonically).
>
> That's because the data is already roughly in PK order.
>
>
> If things are bad enough to require a vacuum full that doesn't seem
> like a good assumption.
>
>
> Sure it does.
>
> For example, I just deleted the oldest half of the records in 30
> tables. Tables who's CREATED_ON timestamp value strongly correlates to
> the synthetic PK sequence values.
>
> Thus, the remaining records were still mostly in PK order. CLUSTERs on
> the PK values would have taken just about as much time as the VACUUM
> FULL statements which I /did/ run.
1) If they are already in enough of a PK order that the CLUSTER time vs
VACUUM FULL time would not be material as there is not much or any
sorting to do then what does the CLUSTER gain you?
Not much. Now they're just "slightly more ordered" instead of "slightly less ordered" for little if any extra effort.
2) What evidence is there that the records where still in PK order just
because you deleted based on CREATED_ON? I understand the correlation
between CREATED_ON and the PK just not sure why that would necessarily
translate to an on disk order by PK?
1. Records are appended to tables in INSERT order, and INSERT order is highly correlated to synthetic PK, by the nature of sequences.
2. My original email showed that CLUSTER took just as long as VACUUM FULL. That means not many records had to be sorted, because... the on-disk order was strongly correlated to PK and CREATED_ON.
Will that happen every time in every circumstance in every database? No, and I never said it would. But it does in my database in this application.
On 4/22/24 12:51, Ron Johnson wrote: > On Mon, Apr 22, 2024 at 3:14 PM Adrian Klaver <adrian.klaver@aklaver.com > <mailto:adrian.klaver@aklaver.com>> wrote: > > > > > 1) If they are already in enough of a PK order that the CLUSTER time vs > VACUUM FULL time would not be material as there is not much or any > sorting to do then what does the CLUSTER gain you? > > > Not much. Now they're just "slightly more ordered" instead of "slightly > less ordered" for little if any extra effort. > > 2) What evidence is there that the records where still in PK order just > because you deleted based on CREATED_ON? I understand the correlation > between CREATED_ON and the PK just not sure why that would necessarily > translate to an on disk order by PK? > > > 1. Records are appended to tables in INSERT order, and INSERT order is > highly correlated to synthetic PK, by the nature of sequences. Not something I would count on, see: https://www.postgresql.org/docs/current/sql-createsequence.html Notes for how that may not always be the case. Also any UPDATE or DELETE is going to change that. There is no guarantee of order for the data in the table. If there where you would not need to run CLUSTER. > 2. My original email showed that CLUSTER took just as long as VACUUM > FULL. That means not many records had to be sorted, because... the > on-disk order was strongly correlated to PK and CREATED_ON. > > Will that happen *every time* in *every circumstance* in *every > database*? No, and I never said it would. But it does in *my *database > in *this *application. > Which gets us back to your comment upstream: "What the VACUUM docs "should" do, it seems, is suggest CLUSTER on the PK, if the PK is a sequence (whether that be an actual sequence, or a timestamp or something else that grows monotonically)." This is a case specific to you and this particular circumstance, not a general rule for VACUUM FULL. If for no other reason then it might make more sense for the application that the CLUSTER be done on some other index then the PK. -- Adrian Klaver adrian.klaver@aklaver.com
On Mon, Apr 22, 2024 at 4:21 PM Adrian Klaver <adrian.klaver@aklaver.com> wrote:
[snip]
Which gets us back to your comment upstream:
"What the VACUUM docs "should" do, it seems, is suggest CLUSTER on the
PK, if the PK is a sequence (whether that be an actual sequence, or a
timestamp or something else that grows monotonically)."
This is a case specific to you and this particular circumstance, not a
general rule for VACUUM FULL. If for no other reason then it might make
more sense for the application that the CLUSTER be done on some other
index then the PK.
On Stack Exchange, I've got a question on how to determine when to run CLUSTER. It ties in strongly with this thread..
On 4/22/24 13:59, Ron Johnson wrote: > On Mon, Apr 22, 2024 at 4:21 PM Adrian Klaver <adrian.klaver@aklaver.com > <mailto:adrian.klaver@aklaver.com>> wrote: > [snip] > > Which gets us back to your comment upstream: > > "What the VACUUM docs "should" do, it seems, is suggest CLUSTER on the > PK, if the PK is a sequence (whether that be an actual sequence, or a > timestamp or something else that grows monotonically)." > > This is a case specific to you and this particular circumstance, not a > general rule for VACUUM FULL. If for no other reason then it might make > more sense for the application that the CLUSTER be done on some other > index then the PK. > > > On Stack Exchange, I've got a question on how to determine when to run > CLUSTER. It ties in strongly with this thread.. > And the link is? -- Adrian Klaver adrian.klaver@aklaver.com
On Mon, Apr 22, 2024 at 5:03 PM Adrian Klaver <adrian.klaver@aklaver.com> wrote:
On 4/22/24 13:59, Ron Johnson wrote:
> On Mon, Apr 22, 2024 at 4:21 PM Adrian Klaver <adrian.klaver@aklaver.com
> <mailto:adrian.klaver@aklaver.com>> wrote:
> [snip]
>
> Which gets us back to your comment upstream:
>
> "What the VACUUM docs "should" do, it seems, is suggest CLUSTER on the
> PK, if the PK is a sequence (whether that be an actual sequence, or a
> timestamp or something else that grows monotonically)."
>
> This is a case specific to you and this particular circumstance, not a
> general rule for VACUUM FULL. If for no other reason then it might make
> more sense for the application that the CLUSTER be done on some other
> index then the PK.
>
>
> On Stack Exchange, I've got a question on how to determine when to run
> CLUSTER. It ties in strongly with this thread..
>
And the link is?
Sorry. Got distracted by the answer.
On 4/22/24 14:35, Ron Johnson wrote: > > > > On Stack Exchange, I've got a question on how to determine when > to run > > CLUSTER. It ties in strongly with this thread.. > > > > And the link is? It should have been the initial question of this thread and it explains what you are really after. > > > Sorry. Got distracted by the answer. > > https://dba.stackexchange.com/questions/338870/when-to-rerun-cluster > <https://dba.stackexchange.com/questions/338870/when-to-rerun-cluster> -- Adrian Klaver adrian.klaver@aklaver.com
On Mon, Apr 22, 2024 at 5:56 PM Adrian Klaver <adrian.klaver@aklaver.com> wrote:
On 4/22/24 14:35, Ron Johnson wrote:
> >
> > On Stack Exchange, I've got a question on how to determine when
> to run
> > CLUSTER. It ties in strongly with this thread..
> >
>
> And the link is?
It should have been the initial question of this thread and it explains
what you are really after.
It was already a long email.