Thread: BUG #2784: Performance serious degrades over a period of a month

The following bug has been logged online:

Bug reference:      2784
Logged by:          Michael Simms
Email address:      michael@tuxgames.com
PostgreSQL version: 8.1.4
Operating system:   Linux kernel 2.6.12
Description:        Performance serious degrades over a period of a month
Details:

OK, we have a database that runs perfectly well after a dump and restore,
but over a period of a month or two, it just degrades to the point of
uselessness.
vacuumdb -a is run every 24 hours. We have also run for months at a time
using -a -z but the effect doesnt change.

The database is for a counter, not the most critical part of the system, but
a part of the system nonetheless. Other tables we have also degrade over
time, but the counter is the most pronounced. There seems to be no common
feature of the tables that degrade. All I know is that a series of queries
that are run on the database every 24 hours, after a dump/restore takes 2
hours. Now, 2 months after, it is taking over 12. We are seriously
considering switching to mysql to avoid this issue.

But I wanted to let you guys have a chance to resolve the issue, we dont
have the manpower or expertise to fix it ourselves. I am willing to let
someone from the postgres development team have access to our server for a
period of time to have a look at the issue. This would need to be someone
extremely trustworthy as the database contains confidential client
information.

I am willing to wait 2 days for a response and for someone to take a look at
the problem. The performance degridation isnt something we can leave as it
is for long, and in 2 days time I will have to dump and restore the
database, which will reset it to a good state, and will mean I will have to
resort to the mysql switch instead.

Sorry this sounds a bit rushed, but it cant be helped, this is causing
*problems* and we need a solution, either a fix or a switch to another
database. Id rather a fix cos I like postgres, but Im willing to bite the
mysql bullet if I have to...

"Michael Simms" <michael@tuxgames.com> writes:
> OK, we have a database that runs perfectly well after a dump and restore,
> but over a period of a month or two, it just degrades to the point of
> uselessness.
> vacuumdb -a is run every 24 hours. We have also run for months at a time
> using -a -z but the effect doesnt change.

You probably need significantly-more-frequent vacuuming.  Have you
considered autovacuum?

This is not a bug --- you'd get better help on the pgsql-performance
mailing list.

            regards, tom lane

This really should have been asked on pgsql-performance and would probably
get a better response there..

On Sun, Nov 26, 2006 at 16:35:52 +0000,
  Michael Simms <michael@tuxgames.com> wrote:
> PostgreSQL version: 8.1.4
> Operating system:   Linux kernel 2.6.12
> Description:        Performance serious degrades over a period of a month
> Details:
>
> OK, we have a database that runs perfectly well after a dump and restore,
> but over a period of a month or two, it just degrades to the point of
> uselessness.
> vacuumdb -a is run every 24 hours. We have also run for months at a time
> using -a -z but the effect doesnt change.
>

This sounds like you either need to increase your FSM setting or vacuum
more often. I think vacuumdb -v will give you enough information to tell
if FSM is too low at the frequency you are vacuuming.

> The database is for a counter, not the most critical part of the system, but
> a part of the system nonetheless. Other tables we have also degrade over
> time, but the counter is the most pronounced. There seems to be no common
> feature of the tables that degrade. All I know is that a series of queries
> that are run on the database every 24 hours, after a dump/restore takes 2
> hours. Now, 2 months after, it is taking over 12. We are seriously
> considering switching to mysql to avoid this issue.

You probably will want to vacuum the counter table more often than the other
tables in the database. Depending on how often the counter(s) are being
updated and how many separate counters are in the table you might want to
vacuum that table as often as once a minute.

Depending on your requirements you might also want to consider using a sequence
instead of a table row for the counter.

Michael Simms wrote:
> The following bug has been logged online:
>
> Bug reference:      2784
> Logged by:          Michael Simms
> Email address:      michael@tuxgames.com
> PostgreSQL version: 8.1.4
> Operating system:   Linux kernel 2.6.12
> Description:        Performance serious degrades over a period of a month
> Details:
>
> OK, we have a database that runs perfectly well after a dump and restore,
> but over a period of a month or two, it just degrades to the point of
> uselessness.
> vacuumdb -a is run every 24 hours. We have also run for months at a time
> using -a -z but the effect doesnt change.

You might have a hung transaction that never finishes open, which
prevents vacuum from removing old tuple versions. Or you might have too
low FSM settings as others suggested.

I'd try running VACUUM VERBOSE by hand, and taking a good look at the
output. If there's nothing obviously wrong with it, please send the
output back to the list (or pgsql-performance, as Tom suggested), and
maybe we can help.

--
   Heikki Linnakangas
   EnterpriseDB   http://www.enterprisedb.com