Re: "Leaking" disk space on FreeBSD servers - Mailing list pgsql-general

On Τετ 20 Μαρ 2013 12:47:39 Dan Thomas wrote:
> > Did you do a detailed du during the supposed problem and after the reboot and make a diff of those to fimd any
invlolvedfiles/dirs? 
>
> du doesn't show the space in question (du -s shows the actual usage on
> disk, df is showing a much higher number), so I doubt this will show
> anything up. However, next reboot I'll certainly do that.

du (without -s)  shows the whole hierarchy, du -s behaves like du -d 0,
so at this point diff the output of (plain) su is definitely somth worth doing.

>
> > That said, i think you might consider posting on freebsd-[questions|stable] as well.
>
> Yes I think that might be a good plan :)
>
> Dan
>
> On 20 March 2013 12:30, Achilleas Mantzios <achill@matrix.gatewaynet.com> wrote:
> > Did you do a detailed du during the supposed problem and after the reboot
> > and make a diff of those
> >
> > to fimd any invlolved files/dirs?
> >
> > That said, i think you might consider posting on freebsd-[questions|stable]
> > as well.
> >
> >
> >
> > On Τετ 20 Μαρ 2013 11:49:07 Dan Thomas wrote:
> >
> > Hi Guys,
> >
> > We're seeing a problem with some of our FreeBSD/PostgreSQL servers "leaking"
> > quite significant amounts of disk space:
> >
> >     > df -h /usr/local/pgsql/
> >     Filesystem       Size    Used   Avail Capacity  Mounted on
> >     /dev/mfid1s1d    1.1T    772G    222G    78%    /usr/local/pgsql
> >
> >     > du -sh /usr/local/pgsql/
> >     741G    /usr/local/pgsql/
> >
> > Stopping Postgres doesn't fix it, but rebooting does which points at the OS
> > rather than PG to me. However, the leak is only apparent in the dedicated
> > pgsql partition, and only on our database servers, so PostgreSQL seems to at
> > least be involved. The partition itself is a relatively standard UFS
> > partition:
> >
> >
> >     > grep /usr/local/pgsql /etc/fstab
> >     /dev/mfid1s1d   /usr/local/pgsql    ufs   rw   2   2
> >
> >     > tunefs -p /usr/local/pgsql/
> >     tunefs: POSIX.1e ACLs: (-a)                                disabled
> >     tunefs: NFSv4 ACLs: (-N)                                   disabled
> >     tunefs: MAC multilabel: (-l)                               disabled
> >     tunefs: soft updates: (-n)                                 enabled
> >     tunefs: gjournal: (-J)                                     disabled
> >     tunefs: trim: (-t)                                         disabled
> >     tunefs: maximum blocks per file in a cylinder group: (-e)  2048
> >     tunefs: average file size: (-f)                            16384
> >     tunefs: average number of files in a directory: (-s)       64
> >     tunefs: minimum percentage of free space: (-m)             8%
> >     tunefs: optimization preference: (-o)                      time
> >     tunefs: volume label: (-L)
> >
> > LSOF isn't showing any open files:
> >
> >     > lsof +L /usr/local/pgsql/ | awk '{ print $8 }' | grep 0 | wc -l
> >     0
> >
> > We're not creating filesystem snapshots:
> >
> >     > find /usr/local/pgsql/ -flags snapshot
> >     >
> >
> > Not all of our servers are leaking space, it's only the more
> > recently-installed systems. Here's a quick breakdown of versions:
> >
> >     FreeBSD   PostgreSQL   Leaking?
> >     8.0       8.4.4        no
> >     8.2       9.0.4        no
> >     8.3       9.1.4        yes
> >     8.3       9.2.3        yes
> >     9.1       9.2.3        yes
> >
> > Each of these servers is configured with a warm standby, so we've been
> > switching them over to the standby to reclaim the space (rebooting the
> > primary is too much downtime). The standby does *not* demonstrate this
> > problem while it's being used as a standby, but it starts leaking space once
> > it's been made the primary.
> >
> > Initially I thought this might be related to WAL files, however the pg_xlog
> > dir is symlinked outside of the /usr/local/pgsql partition that is
> > demonstrating this problem:
> >
> >     > ll /usr/local/pgsql/data/pg_xlog
> >     lrwxr-xr-x 25B Oct 19 10:48 pg_xlog -> /usr/local/pglog/pg_xlog/
> >
> > I've exhausted everything I can think of to try to solve this one. Has
> > anyone got any ideas on how to go about debugging this?
> >
> > Thanks,
> >
> > Dan
> >
> >
> >
> > -
> >
> > Achilleas Mantzios
> >
> > IT DEV
> >
> > IT DEPT
> >
> > Dynacom Tankers Mgmt
>
>
>
-
Achilleas Mantzios
IT DEV
IT DEPT
Dynacom Tankers Mgmt