Thread: Re: [GENERAL] postgres memory management

Re: [GENERAL] postgres memory management

From
Peter Mount
Date:
At 13:18 21/01/01 +0100, Alexander Jerusalem wrote:
>Hi all,
>
>I'm experiencing some strange behaviour with postgresql 7.0.3 on Red Hat
>Linux 7. I'm sending lots of insert statements to the postgresql server
>from another machine via JDBC. During that process postgresql continues to
>take up more and more memory and seemingly never returns it to the system.
>Oddly if I watch the postmaster and it's sub processes in ktop, I can't
>see which process takes up this memory. ktop shows that the postgresql
>related processes have a constant memory usage but the overall memory
>usage always increases as long as I continue to send insert statements.
>
>When the database connection is closed, no memory is reclaimed, the
>overall memory usage stays the same. And when I close down all postgresql
>processes including postmaster, it's the same.
>I'm rather new to Linux and postgresql so I'm not sure if I should call
>this a memory leak :-)
>Has anybody experienced a similar thing?

I'm not sure myself. You can rule out JDBC (or Java) here as you say you
are connecting from another machine.

When your JDBC app closes, does it call the connection's close() method?
Does any messages like "Unexpected EOF from client" appear on the server side?

The only other thing that comes to mine is possibly something weird is
happening with IPC. After you closed down postgres, does ipcclean free up
any memory?

I'm cc'in the hackers list and the new jdbc list.

Peter


>thanks,
>Alexander Jerusalemvknn


Re: Re: [GENERAL] postgres memory management

From
Alfred Perlstein
Date:
* Peter Mount <peter@retep.org.uk> [010122 13:21] wrote:
> At 13:18 21/01/01 +0100, Alexander Jerusalem wrote:
> >Hi all,
> >
> >I'm experiencing some strange behaviour with postgresql 7.0.3 on Red Hat
> >Linux 7. I'm sending lots of insert statements to the postgresql server
> >from another machine via JDBC. During that process postgresql continues to
> >take up more and more memory and seemingly never returns it to the system.
> >Oddly if I watch the postmaster and it's sub processes in ktop, I can't
> >see which process takes up this memory. ktop shows that the postgresql
> >related processes have a constant memory usage but the overall memory
> >usage always increases as long as I continue to send insert statements.
> >
> >When the database connection is closed, no memory is reclaimed, the
> >overall memory usage stays the same. And when I close down all postgresql
> >processes including postmaster, it's the same.
> >I'm rather new to Linux and postgresql so I'm not sure if I should call
> >this a memory leak :-)
> >Has anybody experienced a similar thing?
>
> I'm not sure myself. You can rule out JDBC (or Java) here as you say you
> are connecting from another machine.
>
> When your JDBC app closes, does it call the connection's close() method?
> Does any messages like "Unexpected EOF from client" appear on the server side?
>
> The only other thing that comes to mine is possibly something weird is
> happening with IPC. After you closed down postgres, does ipcclean free up
> any memory?

I don't know if this is valid for Linux, but it is how FreeBSD
works, for the most part used memory is never free'd, it is only
marked as reclaimable.  This is so the system can cache more data.
On a freshly booted FreeBSD box you'll have a lot of 'free' memory,
after the box has been running for a long time the 'free' memory
will probably never go higher that 10megs, the rest is being used
as cache.

The main things you have to worry about is:
a) really running out of memory (are you useing a lot of swap?)
b) not cleaning up IPC as Peter suggested.

--
-Alfred Perlstein - [bright@wintelcom.net|alfred@freebsd.org]
"I have the heart of a child; I keep it in a jar on my desk."

Re: [GENERAL] postgres memory management

From
Alexander Jerusalem
Date:
At 21:40 22.01.01, Peter Mount wrote:
>At 13:18 21/01/01 +0100, Alexander Jerusalem wrote:
>>Hi all,
>>
>>I'm experiencing some strange behaviour with postgresql 7.0.3 on Red Hat
>>Linux 7. I'm sending lots of insert statements to the postgresql server
>>>from another machine via JDBC. During that process postgresql continues to
>>take up more and more memory and seemingly never returns it to the
>>system. Oddly if I watch the postmaster and it's sub processes in ktop, I
>>can't see which process takes up this memory. ktop shows that the
>>postgresql related processes have a constant memory usage but the overall
>>memory usage always increases as long as I continue to send insert statements.
>>
>>When the database connection is closed, no memory is reclaimed, the
>>overall memory usage stays the same. And when I close down all postgresql
>>processes including postmaster, it's the same.
>>I'm rather new to Linux and postgresql so I'm not sure if I should call
>>this a memory leak :-)
>>Has anybody experienced a similar thing?
>
>I'm not sure myself. You can rule out JDBC (or Java) here as you say you
>are connecting from another machine.
>
>When your JDBC app closes, does it call the connection's close() method?
>Does any messages like "Unexpected EOF from client" appear on the server side?
>
>The only other thing that comes to mine is possibly something weird is
>happening with IPC. After you closed down postgres, does ipcclean free up
>any memory?
>
>I'm cc'in the hackers list and the new jdbc list.
>
>Peter

Thanks for your answer!

Yes I'm calling Connection.close(). I don't get any error messages but
maybe I just don't see them because postgresql is started automatically at
run level 3. I'm not sure where the output goes. (pg_log contains only
garbage or maybe it's a binary file) I tried ipcclean right now and it
doesn't free the memory but it gives me some messages that I cannot interpret:

Shared memory 0 ... skipped. Process still exists (pid ).
Shared memory 1 ... skipped. Process still exists (pid ).
Shared memory 2 ... skipped. Process still exists (pid ).
Shared memory 3 ... skipped. Process still exists (pid ).
Semaphore 0 ... resource(s) deleted
Semaphore 1 ... resource(s) deleted

Oddly, when I try to run ipcclean a second time, it says: ipcclean: You
still have a postmaster running. Which is not the case as ps -e proves.

Alexander Jerusalem
ajeru@gmx.net
vknn


Re: Re: postgres memory management

From
Alexander Jerusalem
Date:
At 22:29 22.01.01, Alfred Perlstein wrote:
>* Peter Mount <peter@retep.org.uk> [010122 13:21] wrote:
> > At 13:18 21/01/01 +0100, Alexander Jerusalem wrote:
> > >Hi all,
> > >
> > >I'm experiencing some strange behaviour with postgresql 7.0.3 on Red Hat
> > >Linux 7. I'm sending lots of insert statements to the postgresql server
> > >from another machine via JDBC. During that process postgresql
> continues to
> > >take up more and more memory and seemingly never returns it to the
> system.
> > >Oddly if I watch the postmaster and it's sub processes in ktop, I can't
> > >see which process takes up this memory. ktop shows that the postgresql
> > >related processes have a constant memory usage but the overall memory
> > >usage always increases as long as I continue to send insert statements.
> > >
> > >When the database connection is closed, no memory is reclaimed, the
> > >overall memory usage stays the same. And when I close down all postgresql
> > >processes including postmaster, it's the same.
> > >I'm rather new to Linux and postgresql so I'm not sure if I should call
> > >this a memory leak :-)
> > >Has anybody experienced a similar thing?
> >
> > I'm not sure myself. You can rule out JDBC (or Java) here as you say you
> > are connecting from another machine.
> >
> > When your JDBC app closes, does it call the connection's close() method?
> > Does any messages like "Unexpected EOF from client" appear on the
> server side?
> >
> > The only other thing that comes to mine is possibly something weird is
> > happening with IPC. After you closed down postgres, does ipcclean free up
> > any memory?
>
>I don't know if this is valid for Linux, but it is how FreeBSD
>works, for the most part used memory is never free'd, it is only
>marked as reclaimable.  This is so the system can cache more data.
>On a freshly booted FreeBSD box you'll have a lot of 'free' memory,
>after the box has been running for a long time the 'free' memory
>will probably never go higher that 10megs, the rest is being used
>as cache.
>
>The main things you have to worry about is:
>a) really running out of memory (are you useing a lot of swap?)
>b) not cleaning up IPC as Peter suggested.

Thanks for your answer!

I'm rather new to Linux, so I can't tell if it's that way on Linux. But I
noticed that other programs free some memory when I quit them. But it's
true that I'm not running out of memory. I have 300 MB of free RAM and no
swap space is used. As I wrote in reply to Peters mail, ipcclean doesn't
change anything.

Alexander Jerusalem
ajeru@gmx.net
vknn


Re: postgres memory management

From
Justin Clift
Date:
Hi Alexander,

I've noticed that the PG 7.03 ipcclean script uses "ps x | grep -s
'postmaster'" to determine if a postmaster daemon is still running,
which at least for Mandrake Linux 7.2 doesn't work as expected.  With
this version of linux, the ps & grep combination will find itself and
then ipcclean will complain about an existing postmaster.

I found the solution to this being to edit the ipcclean script and
change the "ps x | grep -s 'postmaster'" part to "ps -e | grep -s
'postmaster'".  This then works correctly with Mandrake 7.2.

Regards and best wishes,

Justin Clift

<snip>
>
> Oddly, when I try to run ipcclean a second time, it says: ipcclean: You
> still have a postmaster running. Which is not the case as ps -e proves.
>
> Alexander Jerusalem
> ajeru@gmx.net
> vknn