Thread: vacuumdb -a -f

vacuumdb -a -f

From
"D Kavan"
Date:
Hello,

Even though I run vacuumdb -a -f every night with no exceptions or problems,
my database size remains 5.6 GB.  After I do a dump/restore, the new
database size is 4.0 GB.  How could that be possible?   That's a significant
amount of space.  If I can clean up my existing databases by 30%, I'd be in
much better shape.

Has anyone else noticed this?

Thanks,
~DjK



Re: vacuumdb -a -f

From
Tom Lane
Date:
"D Kavan" <bitsandbytes88@hotmail.com> writes:
> Even though I run vacuumdb -a -f every night with no exceptions or problems,
> my database size remains 5.6 GB.  After I do a dump/restore, the new
> database size is 4.0 GB.  How could that be possible?

The extra 1.6GB probably represents the amount of junk you generate in
one day.  So, forget the -f and instead do plain vacuums on a more
frequent basis.  Make sure your FSM settings are large enough, too.

            regards, tom lane

Re: vacuumdb -a -f

From
"D Kavan"
Date:
Hi,

Thanks for the tips.

Unfortunatley for me,  even after started doing vacuumdb -a 3 times a day
and increasing fsm dramatically , the size of the database won't go down
even 1 MB.  It's stil at 5.6 GB, size after restore = 4 GB.   I even did a
stop/start instead of a re-load to make sure the settings took affect. Would
a reboot help?

max_fsm_pages = 16000001
max_fsm_relations = 1000000

shared_buffers = 65536
work_mem = 32768
maintenance work mem = 786432

checkpoint_segments = 18


##/etc/sysctl.conf

nel.shmall = 524288
#kernel.shmall = 2097152
#kernel.shmmax = 2147483648
#kernel.shmmax = 1073741824
kernel.shmmax = 6979321856
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
fs.file-max = 65536
net.ipv4.ip_local_port_range = 1024 65000
vm.overcommit_memory = 2


~DjK





>From: Tom Lane <tgl@sss.pgh.pa.us>
>To: "D Kavan" <bitsandbytes88@hotmail.com>
>CC: pgsql-admin@postgresql.org
>Subject: Re: [ADMIN] vacuumdb -a -f Date: Mon, 15 Aug 2005 21:31:01 -0400
>
>"D Kavan" <bitsandbytes88@hotmail.com> writes:
> > Even though I run vacuumdb -a -f every night with no exceptions or
>problems,
> > my database size remains 5.6 GB.  After I do a dump/restore, the new
> > database size is 4.0 GB.  How could that be possible?
>
>The extra 1.6GB probably represents the amount of junk you generate in
>one day.  So, forget the -f and instead do plain vacuums on a more
>frequent basis.  Make sure your FSM settings are large enough, too.
>
>            regards, tom lane



Re: vacuumdb -a -f

From
"Joshua D. Drake"
Date:
D Kavan wrote:
> Hi,
>
> Thanks for the tips.
>
> Unfortunatley for me,  even after started doing vacuumdb -a 3 times a
> day and increasing fsm dramatically , the size of the database won't go
> down even 1 MB.  It's stil at 5.6 GB, size after restore = 4 GB.   I
> even did a stop/start instead of a re-load to make sure the settings
> took affect. Would a reboot help?

A vacuum full will decrease the size but I am not sure why the size is a
problem. I normal vacuum will not remove rows, it only make them
reusable. So at any given point your database is going to grow after use.

I didn't see this whole thread but is there something specific that
seems to  be the issue beyond size?

Sincerely,

Joshua D. Drake




>
> max_fsm_pages = 16000001
> max_fsm_relations = 1000000
>
> shared_buffers = 65536
> work_mem = 32768
> maintenance work mem = 786432
>
> checkpoint_segments = 18
>
>
> ##/etc/sysctl.conf
>
> nel.shmall = 524288
> #kernel.shmall = 2097152
> #kernel.shmmax = 2147483648
> #kernel.shmmax = 1073741824
> kernel.shmmax = 6979321856
> kernel.shmmni = 4096
> kernel.sem = 250 32000 100 128
> fs.file-max = 65536
> net.ipv4.ip_local_port_range = 1024 65000
> vm.overcommit_memory = 2
>
>
> ~DjK
>
>
>
>
>
>> From: Tom Lane <tgl@sss.pgh.pa.us>
>> To: "D Kavan" <bitsandbytes88@hotmail.com>
>> CC: pgsql-admin@postgresql.org
>> Subject: Re: [ADMIN] vacuumdb -a -f Date: Mon, 15 Aug 2005 21:31:01 -0400
>>
>> "D Kavan" <bitsandbytes88@hotmail.com> writes:
>> > Even though I run vacuumdb -a -f every night with no exceptions or
>> problems,
>> > my database size remains 5.6 GB.  After I do a dump/restore, the new
>> > database size is 4.0 GB.  How could that be possible?
>>
>> The extra 1.6GB probably represents the amount of junk you generate in
>> one day.  So, forget the -f and instead do plain vacuums on a more
>> frequent basis.  Make sure your FSM settings are large enough, too.
>>
>>             regards, tom lane
>
>
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 6: explain analyze is your friend


--
Your PostgreSQL solutions company - Command Prompt, Inc. 1.800.492.2240
PostgreSQL Replication, Consulting, Custom Programming, 24x7 support
Managed Services, Shared and Dedicated Hosting
Co-Authors: plPHP, plPerlNG - http://www.commandprompt.com/

Re: vacuumdb -a -f

From
Guido Barosio
Date:
How are you finding out the DB size?
 
G.-

 
On 8/17/05, D Kavan <bitsandbytes88@hotmail.com> wrote:
Hi,

Thanks for the tips.

Unfortunatley for me,  even after started doing vacuumdb -a 3 times a day
and increasing fsm dramatically , the size of the database won't go down
even 1 MB.  It's stil at 5.6 GB, size after restore = 4 GB.   I even did a
stop/start instead of a re-load to make sure the settings took affect. Would
a reboot help?

max_fsm_pages = 16000001
max_fsm_relations = 1000000

shared_buffers = 65536
work_mem = 32768
maintenance work mem = 786432

checkpoint_segments = 18


##/etc/sysctl.conf

nel.shmall = 524288
#kernel.shmall = 2097152
#kernel.shmmax = 2147483648
#kernel.shmmax = 1073741824
kernel.shmmax = 6979321856
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
fs.file-max = 65536
net.ipv4.ip_local_port_range = 1024 65000
vm.overcommit_memory = 2


~DjK





>From: Tom Lane <tgl@sss.pgh.pa.us>
>To: "D Kavan" < bitsandbytes88@hotmail.com>
>CC: pgsql-admin@postgresql.org
>Subject: Re: [ADMIN] vacuumdb -a -f Date: Mon, 15 Aug 2005 21:31:01 -0400
>
>"D Kavan" <bitsandbytes88@hotmail.com> writes:
> > Even though I run vacuumdb -a -f every night with no exceptions or
>problems,
> > my database size remains 5.6 GB.  After I do a dump/restore, the new
> > database size is 4.0 GB.  How could that be possible?
>
>The extra 1.6GB probably represents the amount of junk you generate in
>one day.  So, forget the -f and instead do plain vacuums on a more
>frequent basis.  Make sure your FSM settings are large enough, too.
>
>                       regards, tom lane



---------------------------(end of broadcast)---------------------------
TIP 6: explain analyze is your friend



--
"Adopting the position that you are smarter than an automatic
optimization algorithm is generally a good way to achieve less
performance, not more" - Tom Lane.

Re: vacuumdb -a -f

From
"D Kavan"
Date:

CASE CLOSED!

Thanks for the tip on the reindex script.  That did the trick.  It was one
table in particuliar.

~DjK
















>From: Chris Hoover <revoohc@gmail.com>
>To: Guido Barosio <gbarosio@gmail.com>
>CC: D Kavan <bitsandbytes88@hotmail.com>, pgsql-admin@postgresql.org
>Subject: Re: [ADMIN] vacuumdb -a -f
>Date: Wed, 17 Aug 2005 12:57:16 -0400
>
>You might want to try doing a reindexdb -a -e (reindexdb is in the
>contrib directory of your pg source).  The first time I ran this, I
>gained back a significant amount of space.
>
>I now run a vacuumdb -v -f -a and then a reindexdb -a -e every weekend
>to have PostgreSQL give back as much space as it can.  I have ended up
>doing this do to space and i/o constraints.
>
>Give the reindexdb script a try and see if you don't get your space back.
>
>HTH,
>
>Chris
>
>On 8/17/05, Guido Barosio <gbarosio@gmail.com> wrote:
> > How are you finding out the DB size?
> >
> > G.-
> >
> >
> >
> > On 8/17/05, D Kavan <bitsandbytes88@hotmail.com> wrote:
> > > Hi,
> > >
> > > Thanks for the tips.
> > >
> > > Unfortunatley for me,  even after started doing vacuumdb -a 3 times a
>day
> > > and increasing fsm dramatically , the size of the database won't go
>down
> > > even 1 MB.  It's stil at 5.6 GB, size after restore = 4 GB.   I even
>did a
> > > stop/start instead of a re-load to make sure the settings took affect.
> > Would
> > > a reboot help?
> > >
> > > max_fsm_pages = 16000001
> > > max_fsm_relations = 1000000
> > >
> > > shared_buffers = 65536
> > > work_mem = 32768
> > > maintenance work mem = 786432
> > >
> > > checkpoint_segments = 18
> > >
> > >
> > > ##/etc/sysctl.conf
> > >
> > > nel.shmall = 524288
> > > #kernel.shmall = 2097152
> > > #kernel.shmmax = 2147483648
> > > #kernel.shmmax = 1073741824
> > > kernel.shmmax = 6979321856
> > > kernel.shmmni = 4096
> > > kernel.sem = 250 32000 100 128
> > > fs.file-max = 65536
> > > net.ipv4.ip_local_port_range = 1024 65000
> > > vm.overcommit_memory = 2
> > >
> > >
> > > ~DjK
> > >
> > >
> > >
> > >
> > >
> > > >From: Tom Lane <tgl@sss.pgh.pa.us>
> > > >To: "D Kavan" < bitsandbytes88@hotmail.com>
> > > >CC: pgsql-admin@postgresql.org
> > > >Subject: Re: [ADMIN] vacuumdb -a -f Date: Mon, 15 Aug 2005 21:31:01
>-0400
> > > >
> > > >"D Kavan" <bitsandbytes88@hotmail.com> writes:
> > > > > Even though I run vacuumdb -a -f every night with no exceptions or
> > > >problems,
> > > > > my database size remains 5.6 GB.  After I do a dump/restore, the
>new
> > > > > database size is 4.0 GB.  How could that be possible?
> > > >
> > > >The extra 1.6GB probably represents the amount of junk you generate
>in
> > > >one day.  So, forget the -f and instead do plain vacuums on a more
> > > >frequent basis.  Make sure your FSM settings are large enough, too.
> > > >
> > > >                       regards, tom lane
> > >
> > >
> > >
> > > ---------------------------(end of
> > broadcast)---------------------------
> > > TIP 6: explain analyze is your friend
> > >
> >
> >
> >
> > --
> > "Adopting the position that you are smarter than an automatic
> > optimization algorithm is generally a good way to achieve less
> > performance, not more" - Tom Lane.



Re: vacuumdb -a -f

From
Chris Hoover
Date:
You might want to try doing a reindexdb -a -e (reindexdb is in the
contrib directory of your pg source).  The first time I ran this, I
gained back a significant amount of space.

I now run a vacuumdb -v -f -a and then a reindexdb -a -e every weekend
to have PostgreSQL give back as much space as it can.  I have ended up
doing this do to space and i/o constraints.

Give the reindexdb script a try and see if you don't get your space back.

HTH,

Chris

On 8/17/05, Guido Barosio <gbarosio@gmail.com> wrote:
> How are you finding out the DB size?
>
> G.-
>
>
>
> On 8/17/05, D Kavan <bitsandbytes88@hotmail.com> wrote:
> > Hi,
> >
> > Thanks for the tips.
> >
> > Unfortunatley for me,  even after started doing vacuumdb -a 3 times a day
> > and increasing fsm dramatically , the size of the database won't go down
> > even 1 MB.  It's stil at 5.6 GB, size after restore = 4 GB.   I even did a
> > stop/start instead of a re-load to make sure the settings took affect.
> Would
> > a reboot help?
> >
> > max_fsm_pages = 16000001
> > max_fsm_relations = 1000000
> >
> > shared_buffers = 65536
> > work_mem = 32768
> > maintenance work mem = 786432
> >
> > checkpoint_segments = 18
> >
> >
> > ##/etc/sysctl.conf
> >
> > nel.shmall = 524288
> > #kernel.shmall = 2097152
> > #kernel.shmmax = 2147483648
> > #kernel.shmmax = 1073741824
> > kernel.shmmax = 6979321856
> > kernel.shmmni = 4096
> > kernel.sem = 250 32000 100 128
> > fs.file-max = 65536
> > net.ipv4.ip_local_port_range = 1024 65000
> > vm.overcommit_memory = 2
> >
> >
> > ~DjK
> >
> >
> >
> >
> >
> > >From: Tom Lane <tgl@sss.pgh.pa.us>
> > >To: "D Kavan" < bitsandbytes88@hotmail.com>
> > >CC: pgsql-admin@postgresql.org
> > >Subject: Re: [ADMIN] vacuumdb -a -f Date: Mon, 15 Aug 2005 21:31:01 -0400
> > >
> > >"D Kavan" <bitsandbytes88@hotmail.com> writes:
> > > > Even though I run vacuumdb -a -f every night with no exceptions or
> > >problems,
> > > > my database size remains 5.6 GB.  After I do a dump/restore, the new
> > > > database size is 4.0 GB.  How could that be possible?
> > >
> > >The extra 1.6GB probably represents the amount of junk you generate in
> > >one day.  So, forget the -f and instead do plain vacuums on a more
> > >frequent basis.  Make sure your FSM settings are large enough, too.
> > >
> > >                       regards, tom lane
> >
> >
> >
> > ---------------------------(end of
> broadcast)---------------------------
> > TIP 6: explain analyze is your friend
> >
>
>
>
> --
> "Adopting the position that you are smarter than an automatic
> optimization algorithm is generally a good way to achieve less
> performance, not more" - Tom Lane.

Re: vacuumdb -a -f

From
"D Kavan"
Date:
'du -h' in the base directory.

>How are you finding out the DB size?
>  G.-



Re: vacuumdb -a -f

From
Alvaro Herrera
Date:
On Wed, Aug 17, 2005 at 02:15:42PM -0400, D Kavan wrote:
>
> 'du -h' in the base directory.
>
> >How are you finding out the DB size?

You are considering the difference in XLog segment size, right?  The
pg_xlog directory may start almost empty and then start filling.  The
space will not be recovered by any maintenance command.

--
Alvaro Herrera (<alvherre[a]alvh.no-ip.org>)
Y dijo Dios: "Que sea Satanás, para que la gente no me culpe de todo a mí."
"Y que hayan abogados, para que la gente no culpe de todo a Satanás"