Thread: Memory question
Sorry for posting an obvious Linux question, but have any of you encountered this and how have you fixed it. I have 6gig Ram box. I've set my shmmax to 3072000000. The database starts up fine without any issues. As soon as a query is ran or a FTP process to the server is done, the used memory shoots up and appears to never be released. My fear is that this may cause problems for my database if this number continues to grow. Below is my TOP after running a query, and shutting down PgAdmin. While not low now, the amount of free memory has dropped to around 11mg. I'll admit I'm not that Linux savvy, but am I reading this correct? --TOP 45 processes: 44 sleeping, 1 running, 0 zombie, 0 stopped CPU0 states: 0.0% user, 0.0% system, 0.0% nice, 100.0% idle CPU1 states: 0.0% user, 0.0% system, 0.0% nice, 100.0% idle CPU2 states: 0.0% user, 0.0% system, 0.0% nice, 100.0% idle CPU3 states: 0.0% user, 0.0% system, 0.0% nice, 100.0% idle Mem: 6711564K av, 6517776K used, 193788K free, 0K shrd, 25168K buff Swap: 2044056K av, 0K used, 2044056K free 6257620K cached Patrick Hatcher
This is actually normal. Look at the amount cached: 6257620K. That's 6.2Gig of cache. Linux is using only 6517776k - 6257620k of memory, the rest is just acting as kernel cache. If anything tries to allocate a bit of memory, linux will flush enough cache to give the memory to the application that needs it. Note that you're only showing linux and all its applications using about 256Meg. On Fri, 27 Jun 2003, Patrick Hatcher wrote: > Sorry for posting an obvious Linux question, but have any of you > encountered this and how have you fixed it. > I have 6gig Ram box. I've set my shmmax to 3072000000. The database > starts up fine without any issues. As soon as a query is ran > or a FTP process to the server is done, the used memory shoots up and > appears to never be released. > My fear is that this may cause problems for my database if this number > continues to grow. Below is my TOP after running a query, and shutting > down PgAdmin. While not low now, the amount of free memory has dropped to > around 11mg. I'll admit I'm not that Linux savvy, but am I reading this > correct? > > --TOP > > 45 processes: 44 sleeping, 1 running, 0 zombie, 0 stopped > CPU0 states: 0.0% user, 0.0% system, 0.0% nice, 100.0% idle > CPU1 states: 0.0% user, 0.0% system, 0.0% nice, 100.0% idle > CPU2 states: 0.0% user, 0.0% system, 0.0% nice, 100.0% idle > CPU3 states: 0.0% user, 0.0% system, 0.0% nice, 100.0% idle > Mem: 6711564K av, 6517776K used, 193788K free, 0K shrd, 25168K > buff > Swap: 2044056K av, 0K used, 2044056K free 6257620K > cached > > Patrick Hatcher > > > > ---------------------------(end of broadcast)--------------------------- > TIP 7: don't forget to increase your free space map settings >
Patrick, > Sorry for posting an obvious Linux question, but have any of you > encountered this and how have you fixed it. > I have 6gig Ram box. I've set my shmmax to 3072000000. The database > starts up fine without any issues. As soon as a query is ran > or a FTP process to the server is done, the used memory shoots up and > appears to never be released. What's you shared_buffers set to after our talk? Do you actually need 3gb of shmmax? > My fear is that this may cause problems for my database if this number > continues to grow. Below is my TOP after running a query, and shutting > down PgAdmin. While not low now, the amount of free memory has dropped to > around 11mg. I'll admit I'm not that Linux savvy, but am I reading this > correct? No. > Mem: 6711564K av, 6517776K used, 193788K free, 0K shrd, 25168K The "used" figure in Top doesn't really tell you anything, since it includes the kernel buffer which tries to take up all available memory. If you actually look at the list of processes, I think you'll find that you're only using 1-2% of memory for applications. I'm not sure what app would show your "real" free memory. -- -Josh Berkus Aglio Database Solutions San Francisco
Shared buffer is now set to 20,000 as suggested. So far so good.
As far as shmmax, it really is my ignorance of Linux. We are going to play
around with this number. Is there a suggested amount since I have
my effective_cache_size = 625000 (or does one have nothing to do with the
other)
Thanks again
Patrick Hatcher
Josh Berkus
<josh@agliodbs To: "Patrick Hatcher" <PHatcher@macys.com>,
pgsql-performance@postgresql.org
.com> cc:
Subject: Re: [PERFORM] Memory question
06/27/2003
12:54 PM
Please respond
to josh
Patrick,
> Sorry for posting an obvious Linux question, but have any of you
> encountered this and how have you fixed it.
> I have 6gig Ram box. I've set my shmmax to 3072000000. The database
> starts up fine without any issues. As soon as a query is ran
> or a FTP process to the server is done, the used memory shoots up and
> appears to never be released.
What's you shared_buffers set to after our talk? Do you actually need 3gb
of
shmmax?
> My fear is that this may cause problems for my database if this number
> continues to grow. Below is my TOP after running a query, and shutting
> down PgAdmin. While not low now, the amount of free memory has dropped
to
> around 11mg. I'll admit I'm not that Linux savvy, but am I reading this
> correct?
No.
> Mem: 6711564K av, 6517776K used, 193788K free, 0K shrd, 25168K
The "used" figure in Top doesn't really tell you anything, since it
includes
the kernel buffer which tries to take up all available memory. If you
actually look at the list of processes, I think you'll find that you're
only
using 1-2% of memory for applications.
I'm not sure what app would show your "real" free memory.
--
-Josh Berkus
Aglio Database Solutions
San Francisco
Thank you
Patrick Hatcher
"scott.marlowe
" To: Patrick Hatcher <PHatcher@macys.com>
<scott.marlowe cc: <pgsql-performance@postgresql.org>
@ihs.com> Subject: Re: [PERFORM] Memory question
06/27/2003
12:44 PM
This is actually normal. Look at the amount cached: 6257620K. That's
6.2Gig of cache. Linux is using only 6517776k - 6257620k of memory, the
rest is just acting as kernel cache. If anything tries to allocate a bit
of memory, linux will flush enough cache to give the memory to the
application that needs it.
Note that you're only showing linux and all its applications using about
256Meg.
On Fri, 27 Jun 2003, Patrick Hatcher wrote:
> Sorry for posting an obvious Linux question, but have any of you
> encountered this and how have you fixed it.
> I have 6gig Ram box. I've set my shmmax to 3072000000. The database
> starts up fine without any issues. As soon as a query is ran
> or a FTP process to the server is done, the used memory shoots up and
> appears to never be released.
> My fear is that this may cause problems for my database if this number
> continues to grow. Below is my TOP after running a query, and shutting
> down PgAdmin. While not low now, the amount of free memory has dropped
to
> around 11mg. I'll admit I'm not that Linux savvy, but am I reading this
> correct?
>
> --TOP
>
> 45 processes: 44 sleeping, 1 running, 0 zombie, 0 stopped
> CPU0 states: 0.0% user, 0.0% system, 0.0% nice, 100.0% idle
> CPU1 states: 0.0% user, 0.0% system, 0.0% nice, 100.0% idle
> CPU2 states: 0.0% user, 0.0% system, 0.0% nice, 100.0% idle
> CPU3 states: 0.0% user, 0.0% system, 0.0% nice, 100.0% idle
> Mem: 6711564K av, 6517776K used, 193788K free, 0K shrd, 25168K
> buff
> Swap: 2044056K av, 0K used, 2044056K free 6257620K
> cached
>
> Patrick Hatcher
>
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 7: don't forget to increase your free space map settings
>
On Fri, 2003-06-27 at 12:09, Patrick Hatcher wrote: > I have 6gig Ram box. I've set my shmmax to 3072000000. The database > starts up fine without any issues. As soon as a query is ran > or a FTP process to the server is done, the used memory shoots up and > appears to never be released. In my experience Linux likes to allocate almost all available RAM. I've never had any trouble with that. I'm looking at the memory meter on my RH9 development workstation and it is at 95%. Performance is good, so I just trust that the kernel knows what it is doing. > Mem: 6711564K av, 6517776K used, 193788K free, 0K shrd, 25168K > buff > Swap: 2044056K av, 0K used, 2044056K free 6257620K > cached I've heard anecdotally that Linux has troubles if the swap space is less than the RAM size. I note that you have 6G of RAM, but only 2G of swap. I'm sure others on the list will have more definitive opinions. -- Jord Tanner <jord@indygecko.com>
On Friday, June 27, 2003, at 01:17 PM, Jord Tanner wrote: > On Fri, 2003-06-27 at 12:09, Patrick Hatcher wrote: > > > >> I have 6gig Ram box. I've set my shmmax to 3072000000. The database >> starts up fine without any issues. As soon as a query is ran >> or a FTP process to the server is done, the used memory shoots up and >> appears to never be released. > > In my experience Linux likes to allocate almost all available RAM. I've > never had any trouble with that. I'm looking at the memory meter on my > RH9 development workstation and it is at 95%. Performance is good, so I > just trust that the kernel knows what it is doing. > > > >> Mem: 6711564K av, 6517776K used, 193788K free, 0K shrd, >> 25168K >> buff >> Swap: 2044056K av, 0K used, 2044056K free >> 6257620K >> cached > > I've heard anecdotally that Linux has troubles if the swap space is > less > than the RAM size. I note that you have 6G of RAM, but only 2G of swap. I've heard that too, but it doesn't seem to make much sense to me. If you get to the point where your machine is _needing_ 2GB of swap then something has gone horribly wrong (or you just need more RAM in the machine) and it will just crawl until the kernel kills off whatever process causes the swap space to be exceeded. Seems to me that you should only have that much swap if you can't afford more RAM or you've tapped out your machine's capacity, and your application needs that much memory. -M@
> The "used" figure in Top doesn't really tell you anything,
> since it includes
> the kernel buffer which tries to take up all available
> memory. If you
> actually look at the list of processes, I think you'll find
> that you're only
> using 1-2% of memory for applications.
>
> I'm not sure what app would show your "real" free memory.
The command 'free' shows what you like to know:
$ free
total used free shared buffers
cached
Mem: 1551480 1505656 45824 0 101400
1015540
-/+ buffers/cache: 388716 1162764
Swap: 524264 23088 501176
The used/free amounts on the second line are the interesting ones in
this case.
Arjen
> I've heard that too, but it doesn't seem to make much sense > to me. If > you get to the point where your machine is _needing_ 2GB of swap then > something has gone horribly wrong (or you just need more RAM in the > machine) and it will just crawl until the kernel kills off whatever > process causes the swap space to be exceeded. Seems to me that you > should only have that much swap if you can't afford more RAM > or you've > tapped out your machine's capacity, and your application needs that > much memory. > -M@ I've heard the same, the reason behind it was that there needs to be one-to-one copy of the memory to be able to swap out everything and to have a gain in the total "memory", you'd need twice as much swap as memory to have a doubling of your memory. But afaik this behaviour has been adjusted since the 2.4.5 kernel and isn't a real issue anymore. Please keep in mind that I'm no expert at all on linux, so if you want to be sure, you'd better mail to the kernel-mailinglist orso :) Anyway, I manage a few machines with 1GB++ memory and none of them has more than 1G of swap and none of them uses that swap for more than a few MB unless something was terribly wrong, so the actual 'risk' probably doesn't have a high chance to occur. Arjen
On Fri, 27 Jun 2003, Matthew Hixson wrote: > On Friday, June 27, 2003, at 01:17 PM, Jord Tanner wrote: > > I've heard anecdotally that Linux has troubles if the swap space is > > less > > than the RAM size. I note that you have 6G of RAM, but only 2G of swap. > > I've heard that too, but it doesn't seem to make much sense to me. If > you get to the point where your machine is _needing_ 2GB of swap then > something has gone horribly wrong (or you just need more RAM in the > machine) and it will just crawl until the kernel kills off whatever > process causes the swap space to be exceeded. Seems to me that you > should only have that much swap if you can't afford more RAM or you've > tapped out your machine's capacity, and your application needs that > much memory. This was an artifact in older kernels where the swap code didn't work right unless it had as much swap as memory. I'm pretty sure that was fixed long ago in the 2.4 series.
Arjen van der Meijden wrote: >>I've heard that too, but it doesn't seem to make much sense >>to me. If >>you get to the point where your machine is _needing_ 2GB of swap then >>something has gone horribly wrong (or you just need more RAM in the >>machine) and it will just crawl until the kernel kills off whatever >>process causes the swap space to be exceeded. Seems to me that you >>should only have that much swap if you can't afford more RAM >>or you've >>tapped out your machine's capacity, and your application needs that >>much memory. >> -M@ >> >> >I've heard the same, the reason behind it was that there needs to be >one-to-one copy of the memory to be able to swap out everything and to >have a gain in the total "memory", you'd need twice as much swap as >memory to have a doubling of your memory. > >But afaik this behaviour has been adjusted since the 2.4.5 kernel and >isn't a real issue anymore. > It may be different in vendor released kernels as the default overcommit behavior of the Linux kernel may vary. More detailed discussions can be found on the LKML, or you can find some useful summaries by searching through the last couple "Kernel Traffic" issues <http://kt.zork.net> .. I had some unexpected problems on one system, an older RH distribution, until I actually set the swap to be double the 2GB of ram on the system: 4GB. >Please keep in mind that I'm no expert at all on linux, so if you want >to be sure, you'd better mail to the kernel-mailinglist orso :) > >Anyway, I manage a few machines with 1GB++ memory and none of them has >more than 1G of swap and none of them uses that swap for more than a few >MB unless something was terribly wrong, so the actual 'risk' probably >doesn't have a high chance to occur. > >Arjen > > > > >---------------------------(end of broadcast)--------------------------- >TIP 5: Have you checked our extensive FAQ? > > http://www.postgresql.org/docs/faqs/FAQ.html > >
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Friday 27 June 2003 12:44, scott.marlowe wrote: > This is actually normal. Look at the amount cached: 6257620K. That's > 6.2Gig of cache. Linux is using only 6517776k - 6257620k of memory, the > rest is just acting as kernel cache. If anything tries to allocate a bit > of memory, linux will flush enough cache to give the memory to the > application that needs it. > I think it is appropriate to add that the Linux kernel does this in an extremely innovative and intelligent way. The more room you give your kernel to cache, the more responsive it is going to be. - -- Jonathan Gardner <jgardner@jonathangardner.net> (was jgardn@alumni.washington.edu) Live Free, Use Linux! -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.1 (GNU/Linux) iD8DBQE/AF8EWgwF3QvpWNwRArmVAJwK5C2ExmS8Rayrne33UJ0KZZM4UgCgq7b5 3J1LGtofgtnKq/bPtF75lNI= =4Not -----END PGP SIGNATURE-----