Thread: Libpq memory leak
Program written in C using Libpq, which receives large files (BYTEA) has a memory leak. I need to free ALL of the used memory after each sql query. after each call PQclear() I drop the buffer: conn->inBuffer = realloc(conn->inBuffer, 8192); conn->inBufSize = 8192; It works, but .. I noticed that in some cases PQclear() does not clear the memory. This happens only when the program receives certain files... Maybe there's some buffers that should make realloc()? Or is it a bug? versions tested 9.0.0 and 8.4.4
Polyakov Vladimir <vvpolyakov@gmail.com> writes: > Program written in C using Libpq, which receives large files (BYTEA) > has a memory leak. > I need to free ALL of the used memory after each sql query. > after each call PQclear() I drop the buffer: > conn->inBuffer = realloc(conn->inBuffer, 8192); > conn->inBufSize = 8192; When you break it, you get to keep both pieces. Whatever gave you the idea that the above would be considered a supported thing to do? regards, tom lane
Hey Vladimir,
Regards,
Dmitriy
2010/9/24 Polyakov Vladimir <vvpolyakov@gmail.com>
-- Program written in C using Libpq, which receives large files (BYTEA)
has a memory leak.
I need to free ALL of the used memory after each sql query.
after each call PQclear() I drop the buffer:
conn->inBuffer = realloc(conn->inBuffer, 8192);
conn->inBufSize = 8192;
It works, but ..
I noticed that in some cases PQclear() does not clear the memory.
This happens only when the program receives certain files...
Why do you need realloc() after PQclear()?
Regards,
Dmitriy
On Fri, Sep 24, 2010 at 06:11:31PM +0400, Dmitriy Igrishin wrote: > Hey Vladimir, > > > > 2010/9/24 Polyakov Vladimir <vvpolyakov@gmail.com> > > > Program written in C using Libpq, which receives large files (BYTEA) > > has a memory leak. > > I need to free ALL of the used memory after each sql query. > > > > after each call PQclear() I drop the buffer: > > conn->inBuffer = realloc(conn->inBuffer, 8192); > > conn->inBufSize = 8192; This is a known unsafe use of the realloc() function. If if fails to allocate memory, you just lost the conn->inBuffer, thus leaking memory in your own code. Fix this first, and then see if you still have the issue with memory leaks, because it's possible you're just leaking it with a bad realloc() idiom. > > > > It works, but .. > > I noticed that in some cases PQclear() does not clear the memory. > > This happens only when the program receives certain files... > > > > Why do you need realloc() after PQclear()? > > -- > Regards, > Dmitriy
Tom Lane <tgl@sss.pgh.pa.us> writes: > Polyakov Vladimir <vvpolyakov@gmail.com> writes: > > Program written in C using Libpq, which receives large files (BYTEA) > > has a memory leak. > > I need to free ALL of the used memory after each sql query. I discussed this yesterday with Vladimir. I think Vladimir trying to complain about behavior of libpq input buffer. It is never shrinking, if i not mistaken, it is only grow. So, for example, if you select long bytea field at start of the connection and then select only small fields, you do not need this big inBuffer in memory, but you are still with it during all lifetime of this connection. Here is example: /* -*- mode: c; indent-tabs-mode: nil; c-file-style: "stroustrup"; -*- */ /* * gcc -W -Wall -Wextra -pedantic -o pg-test-bytea-free pg-test-bytea-free.c -lpq * * create table files(id int, data bytea); * insert into files values (60, repeat('1234567890', (60 * 1024 * 1024) / 10)::bytea); * insert into files values (300, repeat('1234567890', (300 * 1024 * 1024) / 10)::bytea); * insert into files values (100, repeat('1234567890', (100 * 1024 * 1024) / 10)::bytea); * */ #include <stdio.h> #include <stdlib.h> #include <libpq-fe.h> #include <internal/libpq-int.h> static void exit_nicely(PGconn *conn) { PQfinish(conn); exit(1); } void test_for(PGconn *conn, int pk) { PGresult *res; char query[128]; int i; sprintf(query, "SELECT data FROM files WHERE id = %d", pk); printf("=== %d ===\n", pk); for (i = 0 ; i < 3; ++i) { res = PQexec(conn, query); if (PQresultStatus(res) != PGRES_TUPLES_OK) { fprintf(stderr, "%s failed: %s", query, PQerrorMessage(conn)); PQclear(res); exit_nicely(conn); } printf("%dMb BEFORE PQclear\n", pk); system("ps axv | grep [pg]-test-bytea"); PQclear(res); printf("%dMb AFTER PQclear\n", pk); system("ps axv | grep [pg]-test-bytea"); printf("*** inBuffer %p inBufSize %10d inStart %10d inCursor %10d inEnd %10d\n", conn->inBuffer, conn->inBufSize, conn->inStart, conn->inCursor, conn->inEnd); puts(""); } } int main () { PGconn *conn; conn = PQconnectdb(""); if (PQstatus(conn) != CONNECTION_OK) { fprintf(stderr, "Connection to database failed: %s", PQerrorMessage(conn)); exit_nicely(conn); } puts("BEFORE PQexec loop"); system("ps axv | grep [pg]-test-bytea"); test_for(conn, 300); test_for(conn, 100); test_for(conn, 60); return 0; } $ gcc -g -g3 -ggdb -ggdb3 -O0 -W -Wall -Wextra -pedantic --std=gnu99 -I/usr/include/postgresql -I/usr/include/postgresql/8.4/server/-lpq pg-test-bytea-free.c -o pg-test-bytea-free $ ./pg-test-bytea-free BEFORE PQexec loop 6533 pts/0 S+ 0:00 0 3 60872 2420 0.0 ./pg-test-bytea-free === 300 === 300Mb BEFORE PQclear 6533 pts/0 S+ 0:00 0 3 892368 616848 15.1 ./pg-test-bytea-free 300Mb AFTER PQclear 6533 pts/0 S+ 0:00 0 3 585164 309644 7.6 ./pg-test-bytea-free *** inBuffer 0x7f2a946cf010 inBufSize 536870912 inStart 314572829 inCursor 314572829 inEnd 314572829 300Mb BEFORE PQclear 6533 pts/0 S+ 0:00 0 3 892368 616848 15.1 ./pg-test-bytea-free 300Mb AFTER PQclear 6533 pts/0 S+ 0:01 0 3 585164 309644 7.6 ./pg-test-bytea-free *** inBuffer 0x7f2a946cf010 inBufSize 536870912 inStart 314572829 inCursor 314572829 inEnd 314572829 300Mb BEFORE PQclear 6533 pts/0 S+ 0:01 0 3 892368 616848 15.1 ./pg-test-bytea-free 300Mb AFTER PQclear 6533 pts/0 S+ 0:01 0 3 585164 309644 7.6 ./pg-test-bytea-free *** inBuffer 0x7f2a946cf010 inBufSize 536870912 inStart 314572829 inCursor 314572829 inEnd 314572829 === 100 === 100Mb BEFORE PQclear 6533 pts/0 S+ 0:01 0 3 687568 412048 10.1 ./pg-test-bytea-free 100Mb AFTER PQclear 6533 pts/0 S+ 0:01 0 3 585164 309644 7.6 ./pg-test-bytea-free *** inBuffer 0x7f2a946cf010 inBufSize 536870912 inStart 104857629 inCursor 104857629 inEnd 104857629 100Mb BEFORE PQclear 6533 pts/0 S+ 0:01 0 3 687568 412048 10.1 ./pg-test-bytea-free 100Mb AFTER PQclear 6533 pts/0 S+ 0:01 0 3 585164 309644 7.6 ./pg-test-bytea-free *** inBuffer 0x7f2a946cf010 inBufSize 536870912 inStart 104857629 inCursor 104857629 inEnd 104857629 100Mb BEFORE PQclear 6533 pts/0 S+ 0:02 0 3 687568 412048 10.1 ./pg-test-bytea-free 100Mb AFTER PQclear 6533 pts/0 S+ 0:02 0 3 585164 309644 7.6 ./pg-test-bytea-free *** inBuffer 0x7f2a946cf010 inBufSize 536870912 inStart 104857629 inCursor 104857629 inEnd 104857629 === 60 === 60Mb BEFORE PQclear 6533 pts/0 S+ 0:02 0 3 646608 371088 9.1 ./pg-test-bytea-free 60Mb AFTER PQclear 6533 pts/0 S+ 0:02 0 3 585164 309644 7.6 ./pg-test-bytea-free *** inBuffer 0x7f2a946cf010 inBufSize 536870912 inStart 62914589 inCursor 62914589 inEnd 62914589 60Mb BEFORE PQclear 6533 pts/0 S+ 0:02 0 3 646608 371088 9.1 ./pg-test-bytea-free 60Mb AFTER PQclear 6533 pts/0 S+ 0:02 0 3 585164 309644 7.6 ./pg-test-bytea-free *** inBuffer 0x7f2a946cf010 inBufSize 536870912 inStart 62914589 inCursor 62914589 inEnd 62914589 60Mb BEFORE PQclear 6533 pts/0 S+ 0:02 0 3 646608 371088 9.1 ./pg-test-bytea-free 60Mb AFTER PQclear 6533 pts/0 S+ 0:02 0 3 585164 309644 7.6 ./pg-test-bytea-free *** inBuffer 0x7f2a946cf010 inBufSize 536870912 inStart 62914589 inCursor 62914589 inEnd 62914589 You are with 309644 RSS at the end. As i can understand Vladimir, he have many clients reading bytea in one server like this and he is trying to reduce memory consumption. -- Sergey Burladyan
Hey Sergey,
--
Regards,
Dmitriy
As i can understand Vladimir, he have many clients reading bytea in one
server like this and he is trying to reduce memory consumption.
Why not use large objects for this purpose?
Regards,
Dmitriy
Hey Sergey,
Why not use large objects for this purpose?
// Dmitriy.
As i can understand Vladimir, he have many clients reading bytea in one
server like this and he is trying to reduce memory consumption.
Why not use large objects for this purpose?
// Dmitriy.