Thread: DROP TABLE and autovacuum

DROP TABLE and autovacuum

From
ITAGAKI Takahiro
Date:
If we tries to drop the table on which autovacuum is running, we have to
wait finish of the vacuum. However, the vacuuming effort goes to waste for
the table being dropped or rewritten. Meanwhile, we've already had the
autovacuum killer triggered in CREATE/DROP/RENAME DATABASE commands.
Can we extend the feature to several TABLE commands?

One simple solution is that every time a non-autovacuum backend tries to
access a table with a lock equal or stronger than SHARE UPDATE EXCLUSIVE,
the backend checks whether some autovacuum workers are vacuuming the table
and send SIGINT to them.

Is this worth doing? Or are there any dangerous situation in it?

Regards,
---
ITAGAKI Takahiro
NTT Open Source Software Center



Re: DROP TABLE and autovacuum

From
Alvaro Herrera
Date:
ITAGAKI Takahiro wrote:
> If we tries to drop the table on which autovacuum is running, we have to
> wait finish of the vacuum. However, the vacuuming effort goes to waste for
> the table being dropped or rewritten. Meanwhile, we've already had the
> autovacuum killer triggered in CREATE/DROP/RENAME DATABASE commands.
> Can we extend the feature to several TABLE commands?
> 
> One simple solution is that every time a non-autovacuum backend tries to
> access a table with a lock equal or stronger than SHARE UPDATE EXCLUSIVE,
> the backend checks whether some autovacuum workers are vacuuming the table
> and send SIGINT to them.
> 
> Is this worth doing? Or are there any dangerous situation in it?

Well, one problem with this is that currently SIGINT cancels the whole
autovacuum worker, not just the table currently being processed.  I
think this can be fixed easily by improving the signal handling.

Aside from that, I don't see any problem in handling DROP TABLE like you
suggest.  But I don't feel comfortable with doing it with just any
strong locker, because that would easily starve tables from being
vacuumed at all.

-- 
Alvaro Herrera                                http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support


Re: DROP TABLE and autovacuum

From
Tom Lane
Date:
ITAGAKI Takahiro <itagaki.takahiro@oss.ntt.co.jp> writes:
> If we tries to drop the table on which autovacuum is running, we have to
> wait finish of the vacuum. However, the vacuuming effort goes to waste for
> the table being dropped or rewritten. Meanwhile, we've already had the
> autovacuum killer triggered in CREATE/DROP/RENAME DATABASE commands.
> Can we extend the feature to several TABLE commands?

> One simple solution is that every time a non-autovacuum backend tries to
> access a table with a lock equal or stronger than SHARE UPDATE EXCLUSIVE,
> the backend checks whether some autovacuum workers are vacuuming the table
> and send SIGINT to them.

I don't think this is a good idea at all.  You're proposing putting a
dangerous sledgehammer into a core part of the system in order to fix a
fairly minor annoyance.

For the specific case of DROP TABLE, a SIGINT might be a good idea
but I don't agree with it for any weaker action.
        regards, tom lane


Re: DROP TABLE and autovacuum

From
ITAGAKI Takahiro
Date:
Alvaro Herrera <alvherre@commandprompt.com> wrote:

> ITAGAKI Takahiro wrote:
> > autovacuum killer triggered in CREATE/DROP/RENAME DATABASE commands.
> > Can we extend the feature to several TABLE commands?
> 
> Well, one problem with this is that currently SIGINT cancels the whole
> autovacuum worker, not just the table currently being processed.  I
> think this can be fixed easily by improving the signal handling.

There is no difference between SIGINT and SIGTERM against autovacuum
workers presently. I'm thinking to split their effects -- SIGINT to 
'skip the current table' and SIGTERM to 'cancel all tables'.

BTW, if autovacuum workers are signaled by an internal server activity,
we will see 'ERROR: canceling statement due to user request' in server log.
Is it surprising to users? I prefer quiet shutdown to ERROR logs.


> Aside from that, I don't see any problem in handling DROP TABLE like you
> suggest.  But I don't feel comfortable with doing it with just any
> strong locker, because that would easily starve tables from being
> vacuumed at all.

Hmm, how about canceling only the cases of DROP TABLE, TRUNCATE and CLUSTER.
We will obviously not need the table after the commands. Other commands,
VACUUM (FULL), ANALYZE, CREATE INDEX (CONCURRENTLY), REINDEX and LOCK TABLE
still conflict with autovacuum, but I'll leave it as-is in the meantime.

Regards,
---
ITAGAKI Takahiro
NTT Open Source Software Center




Re: DROP TABLE and autovacuum

From
Alvaro Herrera
Date:
ITAGAKI Takahiro wrote:
> Alvaro Herrera <alvherre@commandprompt.com> wrote:
> 
> > ITAGAKI Takahiro wrote:
> > > autovacuum killer triggered in CREATE/DROP/RENAME DATABASE commands.
> > > Can we extend the feature to several TABLE commands?
> > 
> > Well, one problem with this is that currently SIGINT cancels the whole
> > autovacuum worker, not just the table currently being processed.  I
> > think this can be fixed easily by improving the signal handling.
> 
> There is no difference between SIGINT and SIGTERM against autovacuum
> workers presently. I'm thinking to split their effects -- SIGINT to 
> 'skip the current table' and SIGTERM to 'cancel all tables'.

Sure, we can do that (it's even mentioned in the comments in autovacuum.c).

> BTW, if autovacuum workers are signaled by an internal server activity,
> we will see 'ERROR: canceling statement due to user request' in server log.
> Is it surprising to users? I prefer quiet shutdown to ERROR logs.

Maybe cancelling the current table processing should be just a WARNING,
not ERROR.  We would abort the transaction quietly.


> > Aside from that, I don't see any problem in handling DROP TABLE like you
> > suggest.  But I don't feel comfortable with doing it with just any
> > strong locker, because that would easily starve tables from being
> > vacuumed at all.
> 
> Hmm, how about canceling only the cases of DROP TABLE, TRUNCATE and CLUSTER.
> We will obviously not need the table after the commands. Other commands,
> VACUUM (FULL), ANALYZE, CREATE INDEX (CONCURRENTLY), REINDEX and LOCK TABLE
> still conflict with autovacuum, but I'll leave it as-is in the meantime.

Well, all of DROP TABLE, TRUNCATE and CLUSTER seem safe -- and also,
they will advance the table's relfrozenxid.  No objection there.

I think all the others you mention should be waiting on autovacuum, not
cancel it.  Maybe what we could do with VACUUM and ANALYZE is let the
user know that the table is being processed by autovacuum and return
quickly.

-- 
Alvaro Herrera                                http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support


Re: DROP TABLE and autovacuum

From
Alvaro Herrera
Date:
Alvaro Herrera wrote:
> ITAGAKI Takahiro wrote:

> > Hmm, how about canceling only the cases of DROP TABLE, TRUNCATE and CLUSTER.
> > We will obviously not need the table after the commands. Other commands,
> > VACUUM (FULL), ANALYZE, CREATE INDEX (CONCURRENTLY), REINDEX and LOCK TABLE
> > still conflict with autovacuum, but I'll leave it as-is in the meantime.
> 
> Well, all of DROP TABLE, TRUNCATE and CLUSTER seem safe -- and also,
> they will advance the table's relfrozenxid.  No objection there.

Something worth considering, though unrelated to the topic at hand: what
happens with the table stats after CLUSTER?  Should we cause an ANALYZE
afterwards?  We could end up running with outdated statistics.

-- 
Alvaro Herrera                                http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support


Re: DROP TABLE and autovacuum

From
ITAGAKI Takahiro
Date:
Alvaro Herrera <alvherre@commandprompt.com> wrote:

> Something worth considering, though unrelated to the topic at hand: what
> happens with the table stats after CLUSTER?  Should we cause an ANALYZE
> afterwards?  We could end up running with outdated statistics.

We don't invalidate the value statistics in pg_stats by ANALYZE presently.

Also, the runtime statistics are not invalidated -- it cound be a bug.
pgstat_drop_relation() is expecting relid (pg_class.oid) as the argument,
but we pass it relfilenode.

[storage/smgr/smgr.c]
static void
smgr_internal_unlink(RelFileNode rnode, int which, bool isTemp, bool isRedo)
{   ...   /*    * Tell the stats collector to forget it immediately, too.    Skip this in    * recovery mode, since the
statscollector likely isn't running (and if    * it is, pgstat.c will get confused because we aren't a real backend
*process).    */   if (!InRecovery)       pgstat_drop_relation(rnode.relNode);
 
   ...
}

Regards,
---
ITAGAKI Takahiro
NTT Open Source Software Center




pgstat_drop_relation bugfix

From
ITAGAKI Takahiro
Date:
I wrote:
> the runtime statistics are not invalidated -- it cound be a bug.
> pgstat_drop_relation() is expecting relid (pg_class.oid) as the argument,
> but we pass it relfilenode.
> 
> [storage/smgr/smgr.c]
> smgr_internal_unlink(RelFileNode rnode, int which, bool isTemp, bool isRedo)
>         pgstat_drop_relation(rnode.relNode);

I'm trying to fix the bug, because there is a possibility that some useless
statistics data continue to occupy some parts of the statistics table.

The bugfix itself is not so difficult; we only need to add a relid field
to PendingRelDelete structure and pass it to pgstat_drop_relation().
However, there are some design issues here.

- smgr need to know relation oid not only relfilenode.  This might brake the module independency.

- What should we do on TRUNCATE, CLUSTER and rewriting table?  The runtime statistics are still valid after those
commands practically, but we discard them in the current logic.  TRUNCATE should be set both n_live_tup and n_dead_tup
tozero.  CLUSTER and rewriting taable should be set only n_dead_tup to zero.  But it might be good to keep other
statistics(# of scans).
 

Are there any other more clever ways?

Regards,
---
ITAGAKI Takahiro
NTT Open Source Software Center