Thread: Re: [ADMIN] WARNING: database must be vacuumed within 8439472 transactions

Re: [ADMIN] WARNING: database must be vacuumed within 8439472 transactions

From

Scott Whitney

Date:

08 July 2014, 00:06:05

Yes. That's what I was referring to. Back in my 8.3 days, I was required to do a FULL vacuum every week. I discussed it quite a bit on this list, and there were many people who said it should not have been required, but it was, and it resolved my particular issues (clogs not getting removed until full vac). I would recommend it.

-------- Original message --------

From: Prabhjot Sheena

Date:07/07/2014 3:46 PM (GMT-06:00)

To: Alvaro Herrera

Cc: pgsql-admin@postgresql.org,Forums postgresql

Subject: Re: [ADMIN] [GENERAL] WARNING: database must be vacuumed within 8439472 transactions

While the vacuumdb --analyze command is running i m getting these messages for these tables which might require full vacuum.

WARNING: relation "public.result" contains more than "max_fsm_pages" pages with useful free space
HINT: Consider using VACUUM FULL on this relation or increasing the configuration parameter "max_fsm_pages".

WARNING: relation "public.run" contains more than "max_fsm_pages" pages with useful free space
HINT: Consider using VACUUM FULL on this relation or increasing the configuration parameter "max_fsm_pages".

On Mon, Jul 7, 2014 at 1:31 PM, Alvaro Herrera <alvherre@2ndquadrant.com> wrote:

Prabhjot Sheena wrote:
> Hello
> We are using postgresql 8.3 database for last 5 yrs for this
> production database and its running fine. This is our critical database
> which runs 24*7. This weekend we started getting these messages
>
> HINT: To avoid a database shutdown, execute a full-database VACUUM.
> WARNING: database must be vacuumed within 8439472 transactions

Did you omit the database name here, or is it really an empty string?
Make sure you vacuum exactly the database mentioned there. Autovacuum
should be doing it, though, but perhaps it's dying for some reason and
it can't vacuum one table in particular. You should check your log for
errors.

--
Álvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

Re: [ADMIN] WARNING: database must be vacuumed within 8439472 transactions

From

Prabhjot Sheena

Date:

08 July 2014, 00:14:57

i will run full vacuum than and see how it goes.

Thanks
avi

On Mon, Jul 7, 2014 at 2:05 PM, Scott Whitney <scott@journyx.com> wrote:

Yes. That's what I was referring to. Back in my 8.3 days, I was required to do a FULL vacuum every week. I discussed it quite a bit on this list, and there were many people who said it should not have been required, but it was, and it resolved my particular issues (clogs not getting removed until full vac). I would recommend it.

-------- Original message --------
From: Prabhjot Sheena
Date:07/07/2014 3:46 PM (GMT-06:00)
To: Alvaro Herrera
Cc: pgsql-admin@postgresql.org,Forums postgresql
Subject: Re: [ADMIN] [GENERAL] WARNING: database must be vacuumed within 8439472 transactions

While the vacuumdb --analyze command is running i m getting these messages for these tables which might require full vacuum.

WARNING: relation "public.result" contains more than "max_fsm_pages" pages with useful free space
HINT: Consider using VACUUM FULL on this relation or increasing the configuration parameter "max_fsm_pages".

WARNING: relation "public.run" contains more than "max_fsm_pages" pages with useful free space

HINT: Consider using VACUUM FULL on this relation or increasing the configuration parameter "max_fsm_pages".

On Mon, Jul 7, 2014 at 1:31 PM, Alvaro Herrera <alvherre@2ndquadrant.com> wrote:
Prabhjot Sheena wrote:
> Hello
> We are using postgresql 8.3 database for last 5 yrs for this
> production database and its running fine. This is our critical database
> which runs 24*7. This weekend we started getting these messages
>
> HINT: To avoid a database shutdown, execute a full-database VACUUM.
> WARNING: database must be vacuumed within 8439472 transactions

Did you omit the database name here, or is it really an empty string?
Make sure you vacuum exactly the database mentioned there. Autovacuum
should be doing it, though, but perhaps it's dying for some reason and
it can't vacuum one table in particular. You should check your log for
errors.

--
Álvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

Re: [ADMIN] WARNING: database must be vacuumed within 8439472 transactions

From

Tom Lane

Date:

08 July 2014, 01:51:28

Prabhjot Sheena <prabhjot.sheena@rivalwatch.com> writes:
> i will run full vacuum than and see how it goes.

I think that is seriously bad advice.  It will take longer and not do
anything more to resolve your immediate problem --- which, it appears,
you don't have a whole lot of time to resolve if you want to avoid a
forced shutdown.

It would likely be worth your time to figure out which table(s) in
which database(s) are actually causing this issue, and vacuum those
first, instead of blindly vacuuming everything.  This will tell you
which database(s) are most problematic:

select datname, age(datfrozenxid) from pg_database order by 2 desc;

and then within those database(s) you can similarly do

select relname, age(relfrozenxid) from pg_class where relkind = 'r' order by 2 desc;

to find the most problematic table(s).

BTW, did you perhaps turn autovacuum off, or cripple its performance
through ill-chosen throttling settings?  It really should've kept you
out of this problem.

            regards, tom lane

Re: [ADMIN] WARNING: database must be vacuumed within 8439472 transactions

From

John R Pierce

Date:

08 July 2014, 01:58:00

On 7/7/2014 2:14 PM, Prabhjot Sheena wrote:
> i will run full vacuum than and see how it goes.

do make sure there aren't any OLD pending transactions hanging around.
if you have any stalled client connections that have left a transaction
open for weeks/months, vacuum can't free any tuples newer than the
oldest transaction.

select * from pg_stat_activity where xact_start < now()-interval '1 hour';

will list all connections with transactions over 1 hour old.







--
john r pierce                                      37N 122W
somewhere on the middle of the left coast

Re: [ADMIN] WARNING: database must be vacuumed within 8439472 transactions

From

Tom Lane

Date:

08 July 2014, 02:15:09

John R Pierce <pierce@hogranch.com> writes:
> On 7/7/2014 2:14 PM, Prabhjot Sheena wrote:
>> i will run full vacuum than and see how it goes.

> do make sure there aren't any OLD pending transactions hanging around.

Not only regular transactions, but prepared transactions:

select * from pg_prepared_xacts;

8.3 was the last release in which max_prepared_transactions was nonzero
by default, thereby allowing people to shoot themselves in the foot
this way without having taken off the safety first :-(

            regards, tom lane

Re: [ADMIN] WARNING: database must be vacuumed within 8439472 transactions

From

Prabhjot Sheena

Date:

08 July 2014, 13:44:40

So this is what i did but my problem is still not going away.

i shutdown the database and started it in single user mode and issued command vacuum full

The command completed but the issue still exists

The thing i noticed is that whenever i start the database autovaccum automatically starts on one table all the time like this
autovacuum: VACUUM public.hotel_site_market (to prevent wraparound)

This same auto vacuum is running since the problem started. i tried to cancel it using pg_cancel_backend but it starts again. i did a vacuum full public.hotel_site_market and the statement completes but again it starts running.

i checked the stats using this

caesius=# select relname, age(relfrozenxid) from pg_class where relkind = 'r' order by 2 desc;
WARNING: database "caesius" must be vacuumed within 1648680 transactions
HINT: To avoid a database shutdown, execute a full-database VACUUM in "caesius".
                      relname                       |    age
----------------------------------------------------+------------
hotel_site_market                                  | 2145834967
cc_table_data                                      |    198017413

Even after running the full vacuum the stats are not changing and this autovacuum: VACUUM public.hotel_site_market (to prevent wraparound) keeps coming back i m getting this message as well

WARNING: database prod01 must be vacuumed within 1648687 transactions

Pls let me know what i should do on this

Thanks

avi

On Mon, Jul 7, 2014 at 4:14 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

John R Pierce <pierce@hogranch.com> writes:
> On 7/7/2014 2:14 PM, Prabhjot Sheena wrote:
>> i will run full vacuum than and see how it goes.

> do make sure there aren't any OLD pending transactions hanging around.

Not only regular transactions, but prepared transactions:

select * from pg_prepared_xacts;

8.3 was the last release in which max_prepared_transactions was nonzero
by default, thereby allowing people to shoot themselves in the foot
this way without having taken off the safety first :-(

regards, tom lane

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [ADMIN] WARNING: database must be vacuumed within 8439472 transactions

From

hubert depesz lubaczewski

Date:

08 July 2014, 14:43:36

First question - are you sure you ran vacuum in the correct database? I.e. in caesius?

Second - is there any long running transaction? select min(xact_start) from pg_stat_activity where xact_start is not null; should tell you.

depesz

On Tue, Jul 8, 2014 at 12:44 PM, Prabhjot Sheena <prabhjot.sheena@rivalwatch.com> wrote:

So this is what i did but my problem is still not going away.

i shutdown the database and started it in single user mode and issued command vacuum full

The command completed but the issue still exists

The thing i noticed is that whenever i start the database autovaccum automatically starts on one table all the time like this
autovacuum: VACUUM public.hotel_site_market (to prevent wraparound)

This same auto vacuum is running since the problem started. i tried to cancel it using pg_cancel_backend but it starts again. i did a vacuum full public.hotel_site_market and the statement completes but again it starts running.

i checked the stats using this

caesius=# select relname, age(relfrozenxid) from pg_class where relkind = 'r' order by 2 desc;
WARNING: database "caesius" must be vacuumed within 1648680 transactions
HINT: To avoid a database shutdown, execute a full-database VACUUM in "caesius".
                      relname                       |    age
----------------------------------------------------+------------
hotel_site_market                                  | 2145834967
cc_table_data                                      |    198017413

Even after running the full vacuum the stats are not changing and this autovacuum: VACUUM public.hotel_site_market (to prevent wraparound) keeps coming back i m getting this message as well

WARNING: database prod01 must be vacuumed within 1648687 transactions

Pls let me know what i should do on this

Thanks
avi

On Mon, Jul 7, 2014 at 4:14 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
John R Pierce <pierce@hogranch.com> writes:
> On 7/7/2014 2:14 PM, Prabhjot Sheena wrote:
>> i will run full vacuum than and see how it goes.

> do make sure there aren't any OLD pending transactions hanging around.

Not only regular transactions, but prepared transactions:

select * from pg_prepared_xacts;

8.3 was the last release in which max_prepared_transactions was nonzero
by default, thereby allowing people to shoot themselves in the foot
this way without having taken off the safety first :-(

regards, tom lane

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [ADMIN] WARNING: database must be vacuumed within 8439472 transactions

From

Prabhjot Sheena

Date:

08 July 2014, 15:39:38

Yes i did ran it in caesius database and not prod01 db that was a typo

there is no long running transactions. i just ran this command select min(xact_start) from pg_stat_activity where xact_start is not null; to make sure

Thanks

On Tue, Jul 8, 2014 at 4:43 AM, hubert depesz lubaczewski <depesz@gmail.com> wrote:

First question - are you sure you ran vacuum in the correct database? I.e. in caesius?

Second - is there any long running transaction? select min(xact_start) from pg_stat_activity where xact_start is not null; should tell you.

depesz

On Tue, Jul 8, 2014 at 12:44 PM, Prabhjot Sheena <prabhjot.sheena@rivalwatch.com> wrote:
So this is what i did but my problem is still not going away.

i shutdown the database and started it in single user mode and issued command vacuum full

The command completed but the issue still exists

The thing i noticed is that whenever i start the database autovaccum automatically starts on one table all the time like this
autovacuum: VACUUM public.hotel_site_market (to prevent wraparound)

This same auto vacuum is running since the problem started. i tried to cancel it using pg_cancel_backend but it starts again. i did a vacuum full public.hotel_site_market and the statement completes but again it starts running.

i checked the stats using this

caesius=# select relname, age(relfrozenxid) from pg_class where relkind = 'r' order by 2 desc;
WARNING: database "caesius" must be vacuumed within 1648680 transactions
HINT: To avoid a database shutdown, execute a full-database VACUUM in "caesius".
                      relname                       |    age
----------------------------------------------------+------------
hotel_site_market                                  | 2145834967
cc_table_data                                      |    198017413

Even after running the full vacuum the stats are not changing and this autovacuum: VACUUM public.hotel_site_market (to prevent wraparound) keeps coming back i m getting this message as well

WARNING: database prod01 must be vacuumed within 1648687 transactions

Pls let me know what i should do on this

Thanks
avi

On Mon, Jul 7, 2014 at 4:14 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
John R Pierce <pierce@hogranch.com> writes:
> On 7/7/2014 2:14 PM, Prabhjot Sheena wrote:
>> i will run full vacuum than and see how it goes.

> do make sure there aren't any OLD pending transactions hanging around.

Not only regular transactions, but prepared transactions:

select * from pg_prepared_xacts;

8.3 was the last release in which max_prepared_transactions was nonzero
by default, thereby allowing people to shoot themselves in the foot
this way without having taken off the safety first :-(

regards, tom lane

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [ADMIN] WARNING: database must be vacuumed within 8439472 transactions

From

hubert depesz lubaczewski

Date:

08 July 2014, 16:12:19

OK. Please run what Tom suggested ( select * from pg_prepared_xacts; ), and show us output.

Also, please run:

vacuum verbose analyze hotel_site_market;

and also show us output.

depesz

On Tue, Jul 8, 2014 at 2:39 PM, Prabhjot Sheena <prabhjot.sheena@rivalwatch.com> wrote:

Yes i did ran it in caesius database and not prod01 db that was a typo

there is no long running transactions. i just ran this command select min(xact_start) from pg_stat_activity where xact_start is not null; to make sure

Thanks

On Tue, Jul 8, 2014 at 4:43 AM, hubert depesz lubaczewski <depesz@gmail.com> wrote:
First question - are you sure you ran vacuum in the correct database? I.e. in caesius?

Second - is there any long running transaction? select min(xact_start) from pg_stat_activity where xact_start is not null; should tell you.

depesz

On Tue, Jul 8, 2014 at 12:44 PM, Prabhjot Sheena <prabhjot.sheena@rivalwatch.com> wrote:
So this is what i did but my problem is still not going away.

i shutdown the database and started it in single user mode and issued command vacuum full

The command completed but the issue still exists

The thing i noticed is that whenever i start the database autovaccum automatically starts on one table all the time like this
autovacuum: VACUUM public.hotel_site_market (to prevent wraparound)

This same auto vacuum is running since the problem started. i tried to cancel it using pg_cancel_backend but it starts again. i did a vacuum full public.hotel_site_market and the statement completes but again it starts running.

i checked the stats using this

caesius=# select relname, age(relfrozenxid) from pg_class where relkind = 'r' order by 2 desc;
WARNING: database "caesius" must be vacuumed within 1648680 transactions
HINT: To avoid a database shutdown, execute a full-database VACUUM in "caesius".
                      relname                       |    age
----------------------------------------------------+------------
hotel_site_market                                  | 2145834967
cc_table_data                                      |    198017413

Even after running the full vacuum the stats are not changing and this autovacuum: VACUUM public.hotel_site_market (to prevent wraparound) keeps coming back i m getting this message as well

WARNING: database prod01 must be vacuumed within 1648687 transactions

Pls let me know what i should do on this

Thanks
avi

On Mon, Jul 7, 2014 at 4:14 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
John R Pierce <pierce@hogranch.com> writes:
> On 7/7/2014 2:14 PM, Prabhjot Sheena wrote:
>> i will run full vacuum than and see how it goes.

> do make sure there aren't any OLD pending transactions hanging around.

Not only regular transactions, but prepared transactions:

select * from pg_prepared_xacts;

8.3 was the last release in which max_prepared_transactions was nonzero
by default, thereby allowing people to shoot themselves in the foot
this way without having taken off the safety first :-(

regards, tom lane

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [ADMIN] WARNING: database must be vacuumed within 8439472 transactions

From

Kevin Grittner

Date:

08 July 2014, 17:22:53

Prabhjot Sheena <prabhjot.sheena@rivalwatch.com> wrote:

> So this is what i did but my problem is still not going away.
>
> i shutdown the database and started it in single user mode and
> issued command vacuum full
>
> The command completed but the issue still exists

That is to be expected.  The VACUUM FULL command is not useful in
this situation.  You need to do a normal VACUUM of the full
database, which is something completely different.  You might want
to read this blog page for a discussion of this:

http://rhaas.blogspot.com/2014/03/vacuum-full-doesnt-mean-vacuum-but.html

If you can upgrade to a supported version you will find many
improvements to VACUUM in general and autovacuum in particular, but
that's probably something to consider after you dig yourself out of
the hole you're in now due to inadequate vacuuming.

--
Kevin Grittner
EDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company