Thread: [CHECKER] 4 memory leaks in Postgresql 7.4.2

[CHECKER] 4 memory leaks in Postgresql 7.4.2

From
Ted Kremenek
Date:
Sorry if this is a repeat.  I checked the mailing list archive and this
message didn't show up, so I am reposting it.

------------------------------------------------------------------------
----------------------------------------

Hi,

I'm from the Stanford Metacompilation research group where we use
static analysis to find bugs.  I'm trying a new technique, so I would
appreciate feedback on these error reports.

This technique tries to infer API roles automatically from the code.
In particular, I am trying to check for ownership properties of
functions (i.e., which functions return ownership of a pointer, which
ones claim it, etc.).  This allows us to potentially find memory leaks
that we didn't check for before because we didn't know what functions
allocated memory or acquired resources.

The tool is still in the early stages, so the bugs reported may be
false positives.  I have applied it to Postgresql 7.4.2.  Included
below are 4 error reports that I believe to be memory leaks.

Any feedback or confirmation of bugs would be appreciated.

Best,
Ted Kremenek

------------------------------------------------------------------------
------------------------------------------

[BUG] memory leak of vacrelstats at end of function laxy_vaccum_rel()
File where bug occurred:
postgresql-7.4.2/src/backend/commands/vacuumlazy.c

   Diagnosis:

       (1) palloc0 returns memory at line 141 and address is stored in
vacrelstats
       (2) vaccrelstats goes out of scope at the end of the function;
leaking the memory


131        BlockNumber possibly_freeable;
132
133        if (vacstmt->verbose)
134            elevel = INFO;
135        else
136            elevel = DEBUG2;
137
138        vacuum_set_xid_limits(vacstmt, onerel->rd_rel->relisshared,
139                              &OldestXmin, &FreezeLimit);
140

    ---> vacrelstats is allocated

141        vacrelstats = (LVRelStats *) palloc0(sizeof(LVRelStats));
142
143        /* Set threshold for interesting free space = average request size
*/
144        /* XXX should we scale it up or down?  Adjust vacuum.c too, if so
*/
145        vacrelstats->threshold = GetAvgFSMRequestSize(&onerel->rd_node);
146
147        /* Open all indexes of the relation */
148        vac_open_indexes(onerel, &nindexes, &Irel);
149        hasindex = (nindexes > 0);
150
151        /* Do the vacuuming */
152        lazy_scan_heap(onerel, vacrelstats, Irel, nindexes);
153
154        /* Done with indexes */
155        vac_close_indexes(nindexes, Irel);
156
157        /*
158         * Optionally truncate the relation.
159         *
160         * Don't even think about it unless we have a shot at releasing a
goodly
161         * number of pages.  Otherwise, the time taken isn't worth it.
162         */
163        possibly_freeable = vacrelstats->rel_pages -
vacrelstats->nonempty_pages;
164        if (possibly_freeable >= REL_TRUNCATE_MINIMUM ||
165         possibly_freeable >= vacrelstats->rel_pages /
REL_TRUNCATE_FRACTION)
166            lazy_truncate_heap(onerel, vacrelstats);
167
168        /* Update shared free space map with final free space info */
169        lazy_update_fsm(onerel, vacrelstats);
170
171        /* Update statistics in pg_class */
172        vac_update_relstats(RelationGetRelid(onerel),
vacrelstats->rel_pages,
173                            vacrelstats->rel_tuples, hasindex);

-->   goes out of scope

174    }

------------------------------------------------------------------------
------------------------------------------

[BUG] memory leak on error path (dtype != DTK_DELTA)
File where bug occurred:
postgresql-7.4.2/src/interfaces/ecpg/pgtypeslib/interval.c

Diagnosis:

    (1) memory is allocated on line 778 by calling pgtypes_alloc; stored
in result
    (2) leaked on error path on line 785

File where bug occurred:
/home/kremenek/oses_src/postgresql-7.4.2/src/interfaces/ecpg/
pgtypeslib/interval.c

768            return NULL;
769        }
770
771        if ((ParseDateTime(str, lowstr, field, ftype, MAXDATEFIELDS, &nf,
ptr) != 0)
772            || (DecodeInterval(field, ftype, nf, &dtype, tm, &fsec) != 0))
773        {
774            errno = PGTYPES_INTVL_BAD_INTERVAL;
775            return NULL;
776        }
777

    ---> memory is allocated


778        result = (interval *) pgtypes_alloc(sizeof(interval));
779        if (!result)
780            return NULL;
781
782        if (dtype != DTK_DELTA)
783        {
784            errno = PGTYPES_INTVL_BAD_INTERVAL;
785            return NULL;

        --->  memory is leaked on error path

786        }
787
788        if (tm2interval(tm, fsec, result) != 0)


------------------------------------------------------------------------
------------------------------------------

[BUG] difficult to analyze, but it appears that subquery is memory
leaked
File where bug occurred:
postgresql-7.4.2/src/backend/optimizer/plan/subselect.c

Diagnosis:

    (1) copyObject() appears to return newly allocated memory on line 261
    (2) subquery is passed to subquery_planner on line 296, but within
that
          function subquery itself never appears to be freed
    (3) subquery has a field dereference on line 301, but afterwards
never
          appears to be referenced
    (4) pointer goes out of scope at end of function

This one I am less confident about, since it is difficult for me to
reason about the inter-procedural semantics.

File where bug occurred:
/home/kremenek/oses_src/postgresql-7.4.2/src/backend/optimizer/plan/
subselect.c

252        int            paramid;
253        List       *lst;
254        Node       *result;
255
256        /*
257         * Copy the source Query node.    This is a quick and dirty kluge to
258         * resolve the fact that the parser can generate trees with
multiple
259         * links to the same sub-Query node, but the planner wants to
scribble
260         * on the Query. Try to clean this up when we do querytree
redesign...
261         */
    --->
262        subquery = (Query *) copyObject(subquery);
263
264        /*
265         * For an EXISTS subplan, tell lower-level planner to expect that
only
266         * the first tuple will be retrieved.  For ALL and ANY subplans, we
267         * will be able to stop evaluating if the test condition fails, so
268         * very often not all the tuples will be retrieved; for lack of a
269         * better idea, specify 50% retrieval.    For EXPR and MULTIEXPR
270         * subplans, use default behavior (we're only expecting one row
out,
271         * anyway).
272         *
273         * NOTE: if you change these numbers, also change
cost_qual_eval_walker()
274         * in path/costsize.c.
275         *
276         * XXX If an ALL/ANY subplan is uncorrelated, we may decide to
hash or
277         * materialize its result below.  In that case it would've been
better
278         * to specify full retrieval.  At present, however, we can only
detect
279         * correlation or lack of it after we've made the subplan :-(.
Perhaps
280         * detection of correlation should be done as a separate step.
281         * Meanwhile, we don't want to be too optimistic about the
percentage
282         * of tuples retrieved, for fear of selecting a plan that's bad for
283         * the materialization case.
284         */
285        if (slink->subLinkType == EXISTS_SUBLINK)
286            tuple_fraction = 1.0;    /* just like a LIMIT 1 */
287        else if (slink->subLinkType == ALL_SUBLINK ||
288                 slink->subLinkType == ANY_SUBLINK)
289            tuple_fraction = 0.5;    /* 50% */
290        else
291            tuple_fraction = 0.0;    /* default behavior */
292
293        /*
294         * Generate the plan for the subquery.
295         */
296        node->plan = plan = subquery_planner(subquery, tuple_fraction);
    --->
297
298        node->plan_id = PlannerPlanId++;    /* Assign unique ID to this
299                                             * SubPlan */
300
301        node->rtable = subquery->rtable;
302
303        /*
304         * Initialize other fields of the SubPlan node.
305         */
306        node->subLinkType = slink->subLinkType;


------------------------------------------------------------------------
------------------------------------------

[BUG] memory leak, pformats is allocated but never freed or stored
anywhere
File where bug occurred: postgresql-7.4.2/src/backend/tcop/postgres.c

Diagnosis:

     (1) pformats is allocated on line 1261
     (2) pformats is never stored anywhere, nor is it freed before going
out of scope

1251        MemoryContextSwitchTo(MessageContext);
1252
1253        /* Get the fixed part of the message */
1254        portal_name = pq_getmsgstring(input_message);
1255        stmt_name = pq_getmsgstring(input_message);
1256
1257        /* Get the parameter format codes */
1258        numPFormats = pq_getmsgint(input_message, 2);
1259        if (numPFormats > 0)
1260        {
    --->
1261            pformats = (int16 *) palloc(numPFormats * sizeof(int16));
    --->
1262            for (i = 0; i < numPFormats; i++)
1263                pformats[i] = pq_getmsgint(input_message, 2);
1264        }
1265
1266        /* Get the parameter value count */
1267        numParams = pq_getmsgint(input_message, 2);
1268
1269        if (numPFormats > 1 && numPFormats != numParams)
1270            ereport(ERROR,
1271                    (errcode(ERRCODE_PROTOCOL_VIOLATION),

Re: [CHECKER] 4 memory leaks in Postgresql 7.4.2

From
Neil Conway
Date:
On 2-May-04, at 2:05 PM, Ted Kremenek wrote:
> I'm from the Stanford Metacompilation research group where we use
> static analysis to find bugs.

Neat. BTW, I saw a talk last summer from Madanlal Musuvathi on some
model checking work which I believe is being done by a related group at
Stanford; it was very interesting.

The problem with applying this kind of static analysis to PostgreSQL is
that palloc() is not like malloc(): if the return value goes out of
scope before it is freed, it is NOT necessarily the case that a memory
leak has occurred. Each palloc() allocation occurs within a "memory
context" (a.k.a an arena, if you're familiar with the usage in tcc).
Individual allocations can be released via pfree(), or the entire
memory context and all memory allocated within it can be released via
MemoryContextReset() or a similar function. Some areas of the code
bother doing a pfree() for each palloc(); some do not.

> [BUG] memory leak of vacrelstats at end of function laxy_vaccum_rel()
> File where bug occurred:
> postgresql-7.4.2/src/backend/commands/vacuumlazy.c

I believe the CommitTransactionCommand() at vacuum.c:894 (which calls
CommitTransaction(), which calls AtCommit_Memory(), which performs a
MemoryContextDelete()) deallocates this memory reasonably soon after it
has been allocated, so this isn't a bug.

> [BUG] memory leak on error path (dtype != DTK_DELTA)
> File where bug occurred:
> postgresql-7.4.2/src/interfaces/ecpg/pgtypeslib/interval.c

Looks suspicious to me, but ECPG is Michael Meskes' domain -- Michael?

> [BUG] difficult to analyze, but it appears that subquery is memory
> leaked
> File where bug occurred:
> postgresql-7.4.2/src/backend/optimizer/plan/subselect.c

Not sure about this one -- I didn't bother tracking down exactly where
the memory context manipulation happens, but I think it's likely that
we release this memory fairly soon after it's allocated.

> [BUG] memory leak, pformats is allocated but never freed or stored
> anywhere

Doesn't look like a bug to me: pformats is allocated in the
MessageContext (e.g. tcop/postgres.c:1308), which is reset for every FE
command that is processed (e.g. postgres.c:2849).

-Neil

Re: [CHECKER] 4 memory leaks in Postgresql 7.4.2

From
Ted Kremenek
Date:
Thanks Neil.  The information about palloc() is extremely useful, and
it will be interesting for us to see how our analysis can better deal
with this.

Thanks for the quick reply!

Best,
Ted

On May 2, 2004, at 12:23 PM, Neil Conway wrote:

> On 2-May-04, at 2:05 PM, Ted Kremenek wrote:
>> I'm from the Stanford Metacompilation research group where we use
>> static analysis to find bugs.
>
> Neat. BTW, I saw a talk last summer from Madanlal Musuvathi on some
> model checking work which I believe is being done by a related group
> at Stanford; it was very interesting.
>
> The problem with applying this kind of static analysis to PostgreSQL
> is that palloc() is not like malloc(): if the return value goes out of
> scope before it is freed, it is NOT necessarily the case that a memory
> leak has occurred. Each palloc() allocation occurs within a "memory
> context" (a.k.a an arena, if you're familiar with the usage in tcc).
> Individual allocations can be released via pfree(), or the entire
> memory context and all memory allocated within it can be released via
> MemoryContextReset() or a similar function. Some areas of the code
> bother doing a pfree() for each palloc(); some do not.
>
>> [BUG] memory leak of vacrelstats at end of function laxy_vaccum_rel()
>> File where bug occurred:
>> postgresql-7.4.2/src/backend/commands/vacuumlazy.c
>
> I believe the CommitTransactionCommand() at vacuum.c:894 (which calls
> CommitTransaction(), which calls AtCommit_Memory(), which performs a
> MemoryContextDelete()) deallocates this memory reasonably soon after
> it has been allocated, so this isn't a bug.
>
>> [BUG] memory leak on error path (dtype != DTK_DELTA)
>> File where bug occurred:
>> postgresql-7.4.2/src/interfaces/ecpg/pgtypeslib/interval.c
>
> Looks suspicious to me, but ECPG is Michael Meskes' domain -- Michael?
>
>> [BUG] difficult to analyze, but it appears that subquery is memory
>> leaked
>> File where bug occurred:
>> postgresql-7.4.2/src/backend/optimizer/plan/subselect.c
>
> Not sure about this one -- I didn't bother tracking down exactly where
> the memory context manipulation happens, but I think it's likely that
> we release this memory fairly soon after it's allocated.
>
>> [BUG] memory leak, pformats is allocated but never freed or stored
>> anywhere
>
> Doesn't look like a bug to me: pformats is allocated in the
> MessageContext (e.g. tcop/postgres.c:1308), which is reset for every
> FE command that is processed (e.g. postgres.c:2849).
>
> -Neil
>

Re: [CHECKER] 4 memory leaks in Postgresql 7.4.2

From
Tom Lane
Date:
Neil Conway <neilc@samurai.com> writes:
> The problem with applying this kind of static analysis to PostgreSQL is
> that palloc() is not like malloc(): if the return value goes out of
> scope before it is freed, it is NOT necessarily the case that a memory
> leak has occurred.

I'm a bit surprised that a tool unaware of this fact would generate only
four complaints ... I'd have expected hundreds.

I concur with Neil's opinion that none of the backend cases represent
bugs.  However:

>> [BUG] memory leak on error path (dtype != DTK_DELTA)
>> File where bug occurred:
>> postgresql-7.4.2/src/interfaces/ecpg/pgtypeslib/interval.c

> Looks suspicious to me, but ECPG is Michael Meskes' domain -- Michael?

It's entirely likely that ecpg's derivative of the backend's datetime
modules contains lots and lots of memory leaks, since AFAIK the palloc
infrastructure is not there in the ecpg environment :-(.

            regards, tom lane

Re: [CHECKER] 4 memory leaks in Postgresql 7.4.2

From
Ted Kremenek
Date:
Since the tool is in its nascent stages, I reported only a handful of
reports that I myself looked at felt that they were potentially bugs.

I appreciate everyone's feedback.

Best,
Ted

On May 2, 2004, at 4:50 PM, Tom Lane wrote:

> Neil Conway <neilc@samurai.com> writes:
>> The problem with applying this kind of static analysis to PostgreSQL
>> is
>> that palloc() is not like malloc(): if the return value goes out of
>> scope before it is freed, it is NOT necessarily the case that a memory
>> leak has occurred.
>
> I'm a bit surprised that a tool unaware of this fact would generate
> only
> four complaints ... I'd have expected hundreds.
>
> I concur with Neil's opinion that none of the backend cases represent
> bugs.  However:
>
>>> [BUG] memory leak on error path (dtype != DTK_DELTA)
>>> File where bug occurred:
>>> postgresql-7.4.2/src/interfaces/ecpg/pgtypeslib/interval.c
>
>> Looks suspicious to me, but ECPG is Michael Meskes' domain -- Michael?
>
> It's entirely likely that ecpg's derivative of the backend's datetime
> modules contains lots and lots of memory leaks, since AFAIK the palloc
> infrastructure is not there in the ecpg environment :-(.
>
>             regards, tom lane
>
> ---------------------------(end of
> broadcast)---------------------------
> TIP 5: Have you checked our extensive FAQ?
>
>                http://www.postgresql.org/docs/faqs/FAQ.html

Re: [CHECKER] 4 memory leaks in Postgresql 7.4.2

From
Michael Meskes
Date:
On Sun, May 02, 2004 at 07:50:46PM -0400, Tom Lane wrote:
> >> [BUG] memory leak on error path (dtype != DTK_DELTA)
> >> File where bug occurred:
> >> postgresql-7.4.2/src/interfaces/ecpg/pgtypeslib/interval.c
>
> > Looks suspicious to me, but ECPG is Michael Meskes' domain -- Michael?
>
> It's entirely likely that ecpg's derivative of the backend's datetime
> modules contains lots and lots of memory leaks, since AFAIK the palloc
> infrastructure is not there in the ecpg environment :-(.

This is true of course. But I'm hopeful we found most of the possible
leaks. This one certainly was not found.

Fix just committed.

Michael
--
Michael Meskes
Email: Michael at Fam-Meskes dot De
ICQ: 179140304, AIM/Yahoo: michaelmeskes, Jabber: meskes@jabber.org
Go SF 49ers! Go Rhein Fire! Use Debian GNU/Linux! Use PostgreSQL!

Re: [CHECKER] 4 memory leaks in Postgresql 7.4.2

From
"Ted Kremenek"
Date:
Thank you for the confirmation.

Best,
Ted

-----Original Message-----
From: Michael Meskes [mailto:meskes@postgresql.org]
Sent: Monday, July 05, 2004 11:55 AM
To: Tom Lane
Cc: Neil Conway; Ted Kremenek; pgsql-bugs@postgresql.org; Michael Meskes
Subject: Re: [BUGS] [CHECKER] 4 memory leaks in Postgresql 7.4.2

On Sun, May 02, 2004 at 07:50:46PM -0400, Tom Lane wrote:
> >> [BUG] memory leak on error path (dtype != DTK_DELTA)
> >> File where bug occurred:
> >> postgresql-7.4.2/src/interfaces/ecpg/pgtypeslib/interval.c
>
> > Looks suspicious to me, but ECPG is Michael Meskes' domain -- Michael?
>
> It's entirely likely that ecpg's derivative of the backend's datetime
> modules contains lots and lots of memory leaks, since AFAIK the palloc
> infrastructure is not there in the ecpg environment :-(.

This is true of course. But I'm hopeful we found most of the possible
leaks. This one certainly was not found.

Fix just committed.

Michael
--
Michael Meskes
Email: Michael at Fam-Meskes dot De
ICQ: 179140304, AIM/Yahoo: michaelmeskes, Jabber: meskes@jabber.org
Go SF 49ers! Go Rhein Fire! Use Debian GNU/Linux! Use PostgreSQL!