Thread: Question about ALTER TABLE DROP CONSTRAINT on 9.1 -- psql crashes

Greg Williamson <gwilliamson39@yahoo.com> writes:
> We have a small database (few hundred megs of data, lass than half that in indexes) that suffers from index bloat.
Currentlywe handle this with an hourly REINDEX command. This works but causes some small issues, so I have been
expploringother methods. 
> When I try to to drop constraints (typically UNIQUE) I sometimes get a crash of the psql client at the ALTER TABLE X
DROPCONSTRAINT, and sometimes after I recreate the constraint at a commit. Transcript below: 

> usher=# BEGIN;
> BEGIN
> gusher=# ALTER TABLE ourcodes DROP CONSTRAINT IF EXISTS ourcodes_pkey CASCADE;
> ALTER TABLE
> gusher=# ALTER TABLE ourcodes ADD CONSTRAINT ourcodes_pkey PRIMARY KEY (id);
> NOTICE:  ALTER TABLE / ADD PRIMARY KEY will create implicit index "ourcodes_pkey" for table "ourcodes"
> ALTER TABLE
> gusher=# COMMIT;
> FATAL:  terminating connection due to administrator command

WTF?  That should certainly not happen.  Have you perhaps got a script
that runs around sending SIGTERM to backends that it thinks are blocking
something?  Does anything show up in the postmaster log when this
happens?

            regards, tom lane

Greg Williamson <gwilliamson39@yahoo.com> writes:
> Every minute on the minute there is a cronjob that kills IDLE IN CONNECTION procs older than 3 seconds ... possibly
thatprocess killed the process I was running, but to me the user it seemed as if the COMMIT came before the disconnect. 

That sure sounds like the source of your problem.  It might be sane if
it killed only processes that *had been idle* for at least three
seconds, but I'm not sure there is any easy way to determine that ...

            regards, tom lane

On 08/10/2012 10:06 AM, Tom Lane wrote:
> Greg Williamson <gwilliamson39@yahoo.com> writes:
>> Every minute on the minute there is a cronjob that kills IDLE IN CONNECTION procs older than 3 seconds ... possibly
thatprocess killed the process I was running, but to me the user it seemed as if the COMMIT came before the disconnect. 
> That sure sounds like the source of your problem.  It might be sane if
> it killed only processes that *had been idle* for at least three
> seconds, but I'm not sure there is any easy way to determine that ...
wouldn't:

select * from pg_stat_activity
where current_query = '<IDLE> in transaction'
AND query_start < current_timestamp - INTERVAL '3 seconds';

do it?

--
Craig Ringer

Craig Ringer <ringerc@ringerc.id.au> writes:
> On 08/10/2012 10:06 AM, Tom Lane wrote:
>> That sure sounds like the source of your problem.  It might be sane if
>> it killed only processes that *had been idle* for at least three
>> seconds, but I'm not sure there is any easy way to determine that ...

> wouldn't:

> select * from pg_stat_activity
> where current_query = '<IDLE> in transaction'
> AND query_start < current_timestamp - INTERVAL '3 seconds';

> do it?

No, that would find sessions that were idle and whose last command
started at least 3 seconds ago.  But it might have completed only
microseconds ago.  The symptoms Greg is describing are consistent
with this kill script running during a short interval between his
index-build command and his COMMIT.

As of 9.2 there's a "state_change" timestamp column in pg_stat_activity
that would provide a safer check, but it's not in existing releases ...

            regards, tom lane

On 08/10/2012 12:20 PM, Tom Lane wrote:
> Craig Ringer <ringerc@ringerc.id.au> writes:
>> On 08/10/2012 10:06 AM, Tom Lane wrote:
>>> That sure sounds like the source of your problem.  It might be sane if
>>> it killed only processes that *had been idle* for at least three
>>> seconds, but I'm not sure there is any easy way to determine that ...
>
>> wouldn't:
>
>> select * from pg_stat_activity
>> where current_query = '<IDLE> in transaction'
>> AND query_start < current_timestamp - INTERVAL '3 seconds';
>
>> do it?
>
> No, that would find sessions that were idle and whose last command
> started at least 3 seconds ago.  But it might have completed only
> microseconds ago.  The symptoms Greg is describing are consistent
> with this kill script running during a short interval between his
> index-build command and his COMMIT.

Eh. For some reason I thought query_start was updated whenever
current_query was set, so it'd be updated when the session went idle.
Nonesensical in retrospect.

Does this approach look sane? Note idlers and then reap them if they
haven't started new queries next pass:

CREATE UNLOGGED TABLE reap (
   procpid integer,
   query_start timestamptz,
   added timestamptz
);

-- Run every 20s or whatever to note idlers and when we noticed them
insert into reap (procpid,query_start,added)
select procpid, query_start, current_timestamp
from pg_stat_activity where current_query = '<IDLE> in transaction';

-- Clean out old entries and kill idlers
WITH r AS (
   DELETE FROM reap r2
   WHERE r2.added + INTERVAL '10 seconds' < current_timestamp
   RETURNING *
)
SELECT r.procpid, pg_terminate_backend(r.procpid)
FROM r INNER JOIN pg_stat_activity s
   ON (r.procpid = s.procpid AND r.query_start = s.query_start )
GROUP BY r.procpid;

ie "where we've noted more than 1 minute ago that a pid was idle, and
query_start hasn't changed so no new query has been run, kill the backend".

Of course, there's race where the backend could be starting a new query
even as you kill it for idleness, but that's true in any
idle-in-transaction control scheme (other than having a true transaction
timeout feature *within* the backend).

--
Craig Ringer