Thread: Re: Allow CLUSTER, VACUUM FULL and REINDEX to change tablespace on the fly

On 2020-12-11 21:27, Alvaro Herrera wrote:
> By the way--  What did you think of the idea of explictly marking the
> types used for bitmasks using types bits32 and friends, instead of plain
> int, which is harder to spot?

If we want to make it clearer, why not turn the thing into a struct, as 
in the attached patch, and avoid the bit fiddling altogether.

On 2020-12-15 03:14, Justin Pryzby wrote:
> On Sat, Dec 12, 2020 at 01:45:26PM -0600, Justin Pryzby wrote:
>> On Sat, Dec 12, 2020 at 09:20:35AM +0100, Peter Eisentraut wrote:
>> > On 2020-12-11 21:27, Alvaro Herrera wrote:
>> > > By the way--  What did you think of the idea of explictly marking the
>> > > types used for bitmasks using types bits32 and friends, instead of plain
>> > > int, which is harder to spot?
>> >
>> > If we want to make it clearer, why not turn the thing into a struct, as in
>> > the attached patch, and avoid the bit fiddling altogether.
>> 
>> I like this.
>> It's a lot like what I wrote as [PATCH v31 1/5] ExecReindex and 
>> ReindexParams
>> In my v31 patch, I moved ReindexOptions to a private structure in 
>> indexcmds.c,
>> with an "int options" bitmask which is passed to reindex_index() et 
>> al.  Your
>> patch keeps/puts ReindexOptions index.h, so it also applies to 
>> reindex_index,
>> which I think is good.
>> 
>> So I've rebased this branch on your patch.
>> 
>> Some thoughts:
>> 
>>  - what about removing the REINDEXOPT_* prefix ?
>>  - You created local vars with initialization like "={}". But I 
>> thought it's
>>    needed to include at least one struct member like "={false}", or 
>> else
>>    they're not guaranteed to be zerod ?
>>  - You passed the structure across function calls.  The usual 
>> convention is to
>>    pass a pointer.
> 
> I think maybe Michael missed this message (?)
> I had applied some changes on top of Peter's patch.
> 
> I squished those commits now, and also handled ClusterOption and 
> VacuumOption
> in the same style.
> 
> Some more thoughts:
>  - should the structures be named in plural ? "ReindexOptions" etc.  
> Since they
>    define *all* the options, not just a single bit.
>  - For vacuum, do we even need a separate structure, or should the 
> members be
>    directly within VacuumParams ?  It's a bit odd to write
>    params.options.verbose.  Especially since there's also ternary 
> options which
>    are directly within params.

This is exactly what I have thought after looking on Peter's patch. 
Writing 'params.options.some_option' looks just too verbose. I even 
started moving all vacuum options into VacuumParams on my own and was in 
the middle of the process when realized that there are some places that 
do not fit well, like:

if (!vacuum_is_relation_owner(RelationGetRelid(onerel),
    onerel->rd_rel,
    params->options & VACOPT_ANALYZE))

Here we do not want to set option permanently, but rather to trigger 
some additional code path in the vacuum_is_relation_owner(), IIUC. With 
unified VacuumParams we should do:

bool opt_analyze = params->analyze;
...
params->analyze = true;
if (!vacuum_is_relation_owner(RelationGetRelid(onerel),
    onerel->rd_rel, params))
...
params->analyze = opt_analyze;

to achieve the same, but it does not look good to me, or change the 
whole interface. I have found at least one other place like that so far 
--- vacuum_open_relation() in the analyze_rel().

Not sure how we could better cope with such logic.

>  - Then, for cluster, I think it should be called ClusterParams, and 
> eventually
>    include the tablespaceOid, like what we're doing for Reindex.
> 
> I am awaiting feedback on these before going further since I've done 
> too much
> rebasing with these ideas going back and forth and back.

Yes, we have moved a bit from my original patch set in the thread with 
all this refactoring. However, all the consequent patches are heavily 
depend on this interface, so let us decide first on the proposed 
refactoring.


Regards
-- 
Alexey Kondratov

Postgres Professional https://www.postgrespro.com
Russian Postgres Company

On 2020-12-23 08:22, Justin Pryzby wrote:
> On Tue, Dec 22, 2020 at 03:22:19PM -0800, Zhihong Yu wrote:
>> Justin:
>> For reindex_index() :
>> 
>> +   if (options->tablespaceOid == MyDatabaseTableSpace)
>> +       options->tablespaceOid = InvalidOid;
>> ...
>> +   oldTablespaceOid = iRel->rd_rel->reltablespace;
>> +   if (set_tablespace &&
>> +       (options->tablespaceOid != oldTablespaceOid ||
>> +       (options->tablespaceOid == MyDatabaseTableSpace &&
>> OidIsValid(oldTablespaceOid))))
>> 
>> I wonder why the options->tablespaceOid == MyDatabaseTableSpace clause
>> appears again in the second if statement.
>> Since the first if statement would assign InvalidOid
>> to options->tablespaceOid when the first if condition is satisfied.
> 
> Good question.  Alexey mentioned on Sept 23 that he added the first 
> stanza.  to
> avoid storing the DB's tablespace OID (rather than InvalidOid).
> 
> I think the 2nd half of the "or" is unnecessary since that was added 
> setting to
> options->tablespaceOid = InvalidOid.
> If requesting to move to the DB's default tablespace, it'll now hit the 
> first
> part of the OR:
> 
>> +       (options->tablespaceOid != oldTablespaceOid ||
> 
> Without the first stanza setting, it would've hit the 2nd condition:
> 
>> +       (options->tablespaceOid == MyDatabaseTableSpace && 
>> OidIsValid(oldTablespaceOid))))
> 
> which means: "user requested to move a table to the DB's default 
> tblspace, and
> it was previously on a nondefault space".
> 
> So I think we can drop the 2nd half of the OR.  Thanks for noticing.

Yes, I have not noticed that we would have already assigned 
tablespaceOid to InvalidOid in this case. Back to the v7 we were doing 
this assignment a bit later, so this could make sense, but now it seems 
to be redundant. For some reason I have mixed these refactorings 
separated by a dozen of versions...


Thanks
-- 
Alexey Kondratov

Postgres Professional https://www.postgrespro.com
Russian Postgres Company

On 2020-12-23 10:38, Michael Paquier wrote:
> On Tue, Dec 22, 2020 at 03:15:37PM -0600, Justin Pryzby wrote:
>> Now, I really think utility.c ought to pass in a pointer to a local
>> ReindexOptions variable to avoid all the memory context, which is 
>> unnecessary
>> and prone to error.
> 
> Yeah, it sounds right to me to just bite the bullet and do this
> refactoring, limiting the manipulations of the options as much as
> possible across contexts.  So +1 from me to merge 0001 and 0002
> together.
> 
> I have adjusted a couple of comments and simplified a bit more the
> code in utility.c.  I think that this is commitable, but let's wait
> more than a couple of days for Alvaro and Peter first.  This is a
> period of vacations for a lot of people, and there is no point to
> apply something that would need more work at the end.  Using hexas for
> the flags with bitmasks is the right conclusion IMO, but we are not
> alone.
> 

After eyeballing the patch I can add that we should alter this comment:

    int    options;    /* bitmask of VacuumOption */

as you are going to replace VacuumOption with VACOPT_* defs. So this 
should say:

/* bitmask of VACOPT_* */

Also I have found naming to be a bit inconsistent:

  * we have ReindexOptions, but VacuumParams
  * and ReindexOptions->flags, but VacuumParams->options

And the last one, you have used bits32 for Cluster/ReindexOptions, but 
left VacuumParams->options as int. Maybe we should also change it to 
bits32 for consistency?


Regards
-- 
Alexey Kondratov

Postgres Professional https://www.postgrespro.com
Russian Postgres Company

On 2021-01-13 14:34, Michael Paquier wrote:
> On Wed, Jan 13, 2021 at 05:22:49PM +0900, Michael Paquier wrote:
>> Yeah, that makes sense.  I'll send an updated patch based on that.
> 
> And here you go as per the attached.  I don't think that there was
> anything remaining on my radar.  This version still needs to be
> indented properly though.
> 
> Thoughts?
> 

Thanks.

+    bits32        options;            /* bitmask of CLUSTEROPT_* */

This should say '/* bitmask of CLUOPT_* */', I guess, since there are 
only CLUOPT's defined. Otherwise, everything looks as per discussed 
upthread.

By the way, something went wrong with the last email subject, so I have 
changed it back to the original in this response. I also attached your 
patch (with only this CLUOPT_* correction) to keep it in the thread for 
sure. Although, postgresql.org's web archive is clever enough to link 
your email to the same thread even with different subject.


Regards
-- 
Alexey Kondratov

Postgres Professional https://www.postgrespro.com
Russian Postgres Company