Thread: file descriptors leak?
Seems like there is (was) a leak of file descriptors somewhere. The descriptors are being used up like crazy. After a week of work on a small database (6 tables, 20 or so indexes) Postgres used up well over 800 descriptors. Is this something known/fixed? [PostgreSQL 6.6.0 on i386-unknown-freebsd3.2, compiled by gcc 2.7.2.1] downloaded and built about the time of 6.5.2 release, sometime in mid-September. Gene Sokolov.
"Gene Sokolov" <hook@aktrad.ru> writes:
> Seems like there is (was) a leak of file descriptors somewhere. The
> descriptors are being used up like crazy.
I fixed some problems along that line during the 6.5 cycle, and thought
the issue closed. Perhaps the problem's come back.
> After a week of work on a small
> database (6 tables, 20 or so indexes) Postgres used up well over 800
> descriptors.
Hmm, there must be multiple descriptors open for the same file then?
That's really weird. Can you obtain a listing of just what is open,
using lsof or some similar tool? Even better, can you provide a
reproducible test case that will cause descriptor leakage?
Also, exactly what do you mean by "Postgres used up..." --- is this
one backend, or a total across the whole system (if so, how many
backends are we talking about here?).
regards, tom lane
From: Tom Lane <tgl@sss.pgh.pa.us> > "Gene Sokolov" <hook@aktrad.ru> writes: > > Seems like there is (was) a leak of file descriptors somewhere. The > > descriptors are being used up like crazy. > > I fixed some problems along that line during the 6.5 cycle, and thought > the issue closed. Perhaps the problem's come back. > > > After a week of work on a small > > database (6 tables, 20 or so indexes) Postgres used up well over 800 > > descriptors. > > Hmm, there must be multiple descriptors open for the same file then? > That's really weird. Can you obtain a listing of just what is open, > using lsof or some similar tool? Even better, can you provide a > reproducible test case that will cause descriptor leakage? We disconnected all clients and the number of descriptors dropped from 800 to about 200, which is reasonable. We currently have 3 connections and ~300 used descriptors. The "lsof -u postgres" is attached. It seems ok except for a large number of open /dev/null. If I hit the problem again, I'll collect the list of open descriptors. > Also, exactly what do you mean by "Postgres used up..." --- is this > one backend, or a total across the whole system (if so, how many > backends are we talking about here?). 1 postmaster, 4-5 backends. If I understand correctly, that is: 1 connection == 1 backend. Gene Sokolov
"Gene Sokolov" <hook@aktrad.ru> writes:
> We disconnected all clients and the number of descriptors dropped from 800
> to about 200, which is reasonable. We currently have 3 connections and ~300
> used descriptors. The "lsof -u postgres" is attached.
Hmm, I see a postmaster with 8 open files and one backend with 34.
Doesn't look out of the ordinary to me.
> It seems ok except for a large number of open /dev/null.
I see /dev/null at the stdin/stdout/stderr positions, which I suppose
means that you started the postmaster with -S instead of directing its
output to a logfile.
It is true that on a system that'll let individual processes have as
many open file descriptors as they want, Postgres can soak up a lot.
Over time I'd expect each backend to acquire an FD for practically
every file in the database directory (including system tables and
indexes). So in a large installation you could be looking at thousands
of open files. But the situation you're describing doesn't seem like
it should reach those kinds of numbers.
The number of open files per backend can be constrained by fd.c, but
AFAIK there isn't any way to set a manually-specified upper limit; it's
all automatic. Perhaps there should be a configuration option to add
a limit.
regards, tom lane
On Tue, 2 Nov 1999, Tom Lane wrote: > Date: Tue, 02 Nov 1999 10:18:15 -0500 > From: Tom Lane <tgl@sss.pgh.pa.us> > To: Gene Sokolov <hook@aktrad.ru> > Cc: pgsql-hackers@postgreSQL.org > Subject: Re: [HACKERS] file descriptors leak? > > "Gene Sokolov" <hook@aktrad.ru> writes: > > We disconnected all clients and the number of descriptors dropped from 800 > > to about 200, which is reasonable. We currently have 3 connections and ~300 > > used descriptors. The "lsof -u postgres" is attached. > > Hmm, I see a postmaster with 8 open files and one backend with 34. > Doesn't look out of the ordinary to me. I see 617 open files (using lsof| grep post | wc). This is a Linux 2.0.37, postgres 6.5.3, 1 postamster and 10 backends. I already complained about this and would glad to understand now is it ok or postgres just wast fd ? > > > It seems ok except for a large number of open /dev/null. > > I see /dev/null at the stdin/stdout/stderr positions, which I suppose > means that you started the postmaster with -S instead of directing its > output to a logfile. In my case most files just /dev/sda..... > > It is true that on a system that'll let individual processes have as > many open file descriptors as they want, Postgres can soak up a lot. > Over time I'd expect each backend to acquire an FD for practically > every file in the database directory (including system tables and > indexes). So in a large installation you could be looking at thousands > of open files. But the situation you're describing doesn't seem like > it should reach those kinds of numbers. > > The number of open files per backend can be constrained by fd.c, but > AFAIK there isn't any way to set a manually-specified upper limit; it's > all automatic. Perhaps there should be a configuration option to add > a limit. > > regards, tom lane > > ************ > _____________________________________________________________ Oleg Bartunov, sci.researcher, hostmaster of AstroNet, Sternberg Astronomical Institute, Moscow University (Russia) Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/ phone: +007(095)939-16-83, +007(095)939-23-83
> On Tue, 2 Nov 1999, Tom Lane wrote: > > > Date: Tue, 02 Nov 1999 10:18:15 -0500 > > From: Tom Lane <tgl@sss.pgh.pa.us> > > To: Gene Sokolov <hook@aktrad.ru> > > Cc: pgsql-hackers@postgreSQL.org > > Subject: Re: [HACKERS] file descriptors leak? > > > > "Gene Sokolov" <hook@aktrad.ru> writes: > > > We disconnected all clients and the number of descriptors dropped from 800 > > > to about 200, which is reasonable. We currently have 3 connections and ~300 > > > used descriptors. The "lsof -u postgres" is attached. > > > > Hmm, I see a postmaster with 8 open files and one backend with 34. > > Doesn't look out of the ordinary to me. > > I see 617 open files (using lsof| grep post | wc). > This is a Linux 2.0.37, postgres 6.5.3, 1 postamster and > 10 backends. I already complained about this and would glad > to understand now is it ok or postgres just wast fd ? PostgreSQL caches up to 64 file descriptors per backend, meaning it keep them open in case it needs them later. The 617 number for 10 backend sounds just about right. -- Bruce Momjian | http://www.op.net/~candle maillist@candle.pha.pa.us | (610) 853-3000+ If your life is a hard drive, | 830 Blythe Avenue + Christ can be your backup. | Drexel Hill, Pennsylvania19026
Oleg Bartunov <oleg@sai.msu.su> writes:
>> Hmm, I see a postmaster with 8 open files and one backend with 34.
>> Doesn't look out of the ordinary to me.
> I see 617 open files (using lsof| grep post | wc).
> This is a Linux 2.0.37, postgres 6.5.3, 1 postamster and
> 10 backends.
About 60 FDs per backend, then. That sounds fairly reasonable to me;
that's probably what fd.c thinks it should limit its usage to on your
platform.
regards, tom lane