Thread: database size grows (even after vacuum (full and analyze))....
Hello all, My subject has been discussed before. I've browsed through the archives and did what I could understand (I'm doing "vaccum analyze" on each of my tables every 30 minutes and "vacuum full" on each of the tables every 2 hour). My database gets about 30 INSERTs per minute, about 60 UPDATEs per minute and about 30 DELETEs per minute in a way that line counts tend to stabilize. The test is done in such a way that I can asure a constant average rate of INSERTs and the same average rate of DELETEs (after a certain life time for each record). Dispite this, the file system size still grows (at about 500KB in about 1000 minutes). I've placed here a pdf file showing my situation. It shows that line counts tend do stabilize. But pg data size slowlly increases (the sudden drops are the result of vacuum full, time unit is 5 minutes). http://robotica.estg.ipvc.pt/software/linux/dbsizes.pdf Can anyone tell me if this is ok, or should I atend to this matter. My system is supposed to work continuouslly without maintenance. Thank you joao
On Wed, May 03, 2006 at 12:02:18 +0100, Joao Miguel Ferreira <jmf@estg.ipvc.pt> wrote: > > My database gets about 30 INSERTs per minute, about 60 UPDATEs per > minute and about 30 DELETEs per minute in a way that line counts tend to > stabilize. The test is done in such a way that I can asure a constant > average rate of INSERTs and the same average rate of DELETEs (after a > certain life time for each record). > > Dispite this, the file system size still grows (at about 500KB in about > 1000 minutes). How often are you vacuuming the table?
On Wed, 2006-05-03 at 15:45, Bruno Wolff III wrote: > On Wed, May 03, 2006 at 12:02:18 +0100, > Joao Miguel Ferreira <jmf@estg.ipvc.pt> wrote: > > certain life time for each record). > > > > Dispite this, the file system size still grows (at about 500KB in about > > 1000 minutes). > > How often are you vacuuming the table? I wrote that on my original e-mail: I do a "VACUUM ANALYZE" every 30 minutes and a "VACUUM FULL" every 2 hours. Should I do anything else ? Thanks. tahnks joao
On Wed, May 03, 2006 at 18:15:37 +0100, Joao Miguel Ferreira <jmf@estg.ipvc.pt> wrote: > On Wed, 2006-05-03 at 15:45, Bruno Wolff III wrote: > > On Wed, May 03, 2006 at 12:02:18 +0100, > > Joao Miguel Ferreira <jmf@estg.ipvc.pt> wrote: > > > certain life time for each record). > > > > > > Dispite this, the file system size still grows (at about 500KB in about > > > 1000 minutes). > > > > How often are you vacuuming the table? > > I wrote that on my original e-mail: I do a "VACUUM ANALYZE" every 30 > minutes and a "VACUUM FULL" every 2 hours. Sorry about that, I obviously missed that. > Should I do anything else ? Thanks. There are a couple of possibilities worth checking. One is that there aren't idle transactions staying open for a long time. These would prevent vacuum from removing deleted rows as these transactions could still see them. Another possibility is that the FSM is too low and there isn't enough space to track all of rows that can be recovered. (Vacuuming more often will also reduce the needed FSM setting.) A third possible issue is index bloat, which can happen on older versions (7.4ish) of Postgres where key values increase (or decrease) montonicly.
> There are a couple of possibilities worth checking. One is that there aren't > idle transactions staying open for a long time. These would prevent vacuum > from removing deleted rows as these transactions could still see them. bruno, thanks for the feedback. some things; 1) How can I check these possible open transactions ? > Another possibility is that the FSM is too low and there isn't enough space to > track all of rows that can be recovered. (Vacuuming more often will also > reduce the needed FSM setting.) A third possible issue is index bloat, which > can happen on older versions (7.4ish) of Postgres where key values increase (or > decrease) montonicly. 2) what is FSM ? 3) mybe you are right. one of the tables has a UNIQUE clause that includes an ever growing INT column (the UNIQUE clause also includes some TEXT columns). The other two tables also include one column that is an ever growing INT, too, with no special clauses. But I'm using Pg7.2 (as originally included in Fedora Core 3). Is this something to care about or should I simply let it be ? Thanks joao
Joao Miguel Ferreira <jmf@estg.ipvc.pt> writes:
> On Wed, 2006-05-03 at 15:45, Bruno Wolff III wrote:
>> How often are you vacuuming the table?
> I wrote that on my original e-mail: I do a "VACUUM ANALYZE" every 30
> minutes and a "VACUUM FULL" every 2 hours.
That's not a good maintenance procedure. Just do the plain vacuums and
forget the VACUUM FULL. If the table row count is fairly stable then
plain vacuums are all you need. I suspect VACUUM FULL is making things
worse not better --- it'll compact the tables, but at the price of
bloating the indexes.
You might want to pay some attention to exactly where the space is
disappearing to --- which files are getting bigger?
If you find you are suffering from index bloat, a very occasional
REINDEX (maybe once a week or less) will fix that, but VACUUM FULL
won't help it at all. I wouldn't recommend doing this unless
proven necessary, however.
regards, tom lane
ok, I understand that. Thank you. I'll try it. joao On Wed, 2006-05-03 at 19:12, Tom Lane wrote: > Joao Miguel Ferreira <jmf@estg.ipvc.pt> writes: > > On Wed, 2006-05-03 at 15:45, Bruno Wolff III wrote: > >> How often are you vacuuming the table? > > > I wrote that on my original e-mail: I do a "VACUUM ANALYZE" every 30 > > minutes and a "VACUUM FULL" every 2 hours. > > That's not a good maintenance procedure. Just do the plain vacuums and > forget the VACUUM FULL. If the table row count is fairly stable then > plain vacuums are all you need. I suspect VACUUM FULL is making things > worse not better --- it'll compact the tables, but at the price of > bloating the indexes. > > You might want to pay some attention to exactly where the space is > disappearing to --- which files are getting bigger? > > If you find you are suffering from index bloat, a very occasional > REINDEX (maybe once a week or less) will fix that, but VACUUM FULL > won't help it at all. I wouldn't recommend doing this unless > proven necessary, however. > > regards, tom lane thx joao
On Wed, May 03, 2006 at 18:57:24 +0100, Joao Miguel Ferreira <jmf@estg.ipvc.pt> wrote: > > 1) How can I check these possible open transactions ? http://developer.postgresql.org/docs/postgres/monitoring-ps.html > 2) what is FSM ? http://developer.postgresql.org/docs/postgres/runtime-config-resource.html#RUNTIME-CONFIG-RESOURCE-FSM
Hi Tom and all, > I wrote that on my original e-mail: I do a "VACUUM ANALYZE" every 30 > > minutes and a "VACUUM FULL" every 2 hours. > > That's not a good maintenance procedure. Just do the plain vacuums and > forget the VACUUM FULL. If the table row count is fairly stable then > plain vacuums are all you need. I suspect VACUUM FULL is making things > worse not better --- it'll compact the tables, but at the price of > bloating the indexes. > > You might want to pay some attention to exactly where the space is > disappearing to --- which files are getting bigger? Sorry for the long silence. I've been making experiments and reading about all of this. The 'data' directory is the one growing in size. > > If you find you are suffering from index bloat, a very occasional > REINDEX (maybe once a week or less) will fix that, but VACUUM FULL > won't help it at all. I wouldn't recommend doing this unless > proven necessary, however. I've checked (and read about) REINDEX and applied it. Nevertheless I still get a litlle continuous growth (about 200Bytes/minute) in the 'data' directory. I must say: REINDEX causes a sudden drop in filesystem usage but (about 30 minutes later) size returns to where it droped and continues growing. What would you say I'dd do next ? thx jmf PS: this problem brought me to an excelent book about Pg. I'm suprised with the great things I didn't know about Pg... > > regards, tom lane