Thread: Re: [HACKERS] Everything leaks; How it mm suppose to work?
> >Running postgresql in interactive mode shows that for each query I >type there is memory lost. The exact amount of memory lost depends on >the query I use. The amount of memory not freed is also a function >of the number of tuples returned. > Oops, it seems some palloced memory is not freed by pfree but using some other function(s). My mistake, sorry. Thanks, with regards from Maurice.
>
>
> >
> >Running postgresql in interactive mode shows that for each query I
> >type there is memory lost. The exact amount of memory lost depends on
> >the query I use. The amount of memory not freed is also a function
> >of the number of tuples returned.
> >
>
> Oops, it seems some palloced memory is not freed by pfree but
> using some other function(s).
> My mistake, sorry.
>
One thing I have found is that:
select * from test where 1=0;
do not leak memory while
select * from test where x=-999;
does leak memory, even though neither returns any rows. Strange. Would
seem to point to the optimizer or executor.
--
Bruce Momjian | 830 Blythe Avenue
maillist@candle.pha.pa.us | Drexel Hill, Pennsylvania 19026
+ If your life is a hard drive, | (610) 353-9879(w)
+ Christ can be your backup. | (610) 853-3000(h)
>
>
> >
> >Running postgresql in interactive mode shows that for each query I
> >type there is memory lost. The exact amount of memory lost depends on
> >the query I use. The amount of memory not freed is also a function
> >of the number of tuples returned.
> >
>
> Oops, it seems some palloc'ed memory is not freed by pfree() but
> using some other function(s).
> My mistake, sorry.
OK, I think I know where the leak is from. I checked AllocSetDump(), and
it did not show any problems with any context growing, so I started to
suspect direct calls to malloc(). I tried trapping on malloc(), but
there are too many calls.
Then I ran profiling on the two queries I mentioned, where one leaks and
one doesn't, and found that the leaking one had 500 extra calls to
malloc. Grep'ing out the calls and comparing the output of the two
profiles, I found:
0.00 0.00 1/527 ___irs_gen_acc [162]
0.00 0.00 35/527 _GetDatabasePath [284]
0.00 0.00 491/527 _GetDatabaseName [170]
[166] 0.8 0.00 0.00 527 _strdup [166]
0.00 0.00 527/2030 _malloc [105]
0.00 0.00 527/604 _strlen [508]
0.00 0.00 527/532 _bcopy [515]
I believe this code was modified by Vadim to fix our problem with blind
write errors when using psql while the regression tests were being run.
Am I correct on this? I have not developed a fix yet.
--
Bruce Momjian | 830 Blythe Avenue
maillist@candle.pha.pa.us | Drexel Hill, Pennsylvania 19026
+ If your life is a hard drive, | (610) 353-9879(w)
+ Christ can be your backup. | (610) 853-3000(h)
Bruce Momjian wrote: > > Then I ran profiling on the two queries I mentioned, where one leaks and > one doesn't, and found that the leaking one had 500 extra calls to > malloc. Grep'ing out the calls and comparing the output of the two > profiles, I found: > > 0.00 0.00 1/527 ___irs_gen_acc [162] > 0.00 0.00 35/527 _GetDatabasePath [284] > 0.00 0.00 491/527 _GetDatabaseName [170] > [166] 0.8 0.00 0.00 527 _strdup [166] > 0.00 0.00 527/2030 _malloc [105] > 0.00 0.00 527/604 _strlen [508] > 0.00 0.00 527/532 _bcopy [515] > > I believe this code was modified by Vadim to fix our problem with blind > write errors when using psql while the regression tests were being run. > > Am I correct on this? I have not developed a fix yet. Only src/backend/storage/smgr/md.c:mdblindwrt() was changed... Vadim
Bruce: > Maurice: > > >Running postgresql in interactive mode shows that for each query I > > >type there is memory lost. The exact amount of memory lost depends on > > >the query I use. The amount of memory not freed is also a function > > >of the number of tuples returned. > > > > > > > Oops, it seems some palloced memory is not freed by pfree but > > using some other function(s). > > My mistake, sorry. > > > > One thing I have found is that: > > select * from test where 1=0; > > do not leak memory while > > select * from test where x=-999; > > does leak memory, even though neither returns any rows. Strange. Would > seem to point to the optimizer or executor. In the first case, the where clause is constant and evaluates false, so not much is done. In the second case, the table is scanned and presumably some memory is allocated for each row, probably to evaluate the expression. Since this memory is allocated into a per statement duration, it will not be freed until the end of the statement when the context is destroyed. -dg David Gould dg@illustra.com 510.628.3783 or 510.305.9468 Informix Software (No, really) 300 Lakeside Drive Oakland, CA 94612 - Linux. Not because it is free. Because it is better.
> > Bruce: > > Maurice: > > > >Running postgresql in interactive mode shows that for each query I > > > >type there is memory lost. The exact amount of memory lost depends on > > > >the query I use. The amount of memory not freed is also a function > > > >of the number of tuples returned. > > > > > > > > > > Oops, it seems some palloced memory is not freed by pfree but > > > using some other function(s). > > > My mistake, sorry. > > > > > > > One thing I have found is that: > > > > select * from test where 1=0; > > > > do not leak memory while > > > > select * from test where x=-999; > > > > does leak memory, even though neither returns any rows. Strange. Would > > seem to point to the optimizer or executor. > > In the first case, the where clause is constant and evaluates false, so > not much is done. In the second case, the table is scanned and presumably > some memory is allocated for each row, probably to evaluate the expression. > Since this memory is allocated into a per statement duration, it will not > be freed until the end of the statement when the context is destroyed. > > -dg > Does it make sense to have a 'row' context which is released just before starting with a new tuple ? The total number or free is the same but they are distributed over the query and unused memory should not accumulate. I have seen backends growing to 40-60MB with queries which scan a very large number of rows. Massimo Dal Zotto +----------------------------------------------------------------------+ | Massimo Dal Zotto e-mail: dz@cs.unitn.it | | Via Marconi, 141 phone: ++39-461-534251 | | 38057 Pergine Valsugana (TN) www: http://www.cs.unitn.it/~dz/ | | Italy pgp: finger dz@tango.cs.unitn.it | +----------------------------------------------------------------------+
> > Bruce: > > Maurice: > > > >Running postgresql in interactive mode shows that for each query I > > > >type there is memory lost. The exact amount of memory lost depends on > > > >the query I use. The amount of memory not freed is also a function > > > >of the number of tuples returned. > > > > > > > > > > Oops, it seems some palloced memory is not freed by pfree but > > > using some other function(s). > > > My mistake, sorry. > > > > > > > One thing I have found is that: > > > > select * from test where 1=0; > > > > do not leak memory while > > > > select * from test where x=-999; > > > > does leak memory, even though neither returns any rows. Strange. Would > > seem to point to the optimizer or executor. > > In the first case, the where clause is constant and evaluates false, so > not much is done. In the second case, the table is scanned and presumably > some memory is allocated for each row, probably to evaluate the expression. > Since this memory is allocated into a per statement duration, it will not > be freed until the end of the statement when the context is destroyed. This will be fixed in the 6.3.2 patch, due out soon. -- Bruce Momjian | 830 Blythe Avenue maillist@candle.pha.pa.us | Drexel Hill, Pennsylvania 19026 + If your life is a hard drive, | (610) 353-9879(w) + Christ can be your backup. | (610) 853-3000(h)