Thread: Memory usage of writer process
The writer process seems to be using inordinate amounts of memory: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 11088 postgres 13 -2 3217m 2.9g 2.9g S 0 38.7 0:10.46 postgres: writer process 20190 postgres 13 -2 3219m 71m 68m S 0 0.9 0:52.48 postgres: cribq cribq [local] idle I am writing moderately large (~3k) records to my database a few times a second. Even when I stop doing that, the process continues to take up all of that memory. Am I reading this right? Why is it using so much memory?
Alex wrote: > The writer process seems to be using inordinate amounts of memory: > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ > COMMAND > 11088 postgres 13 -2 3217m 2.9g 2.9g S 0 38.7 0:10.46 postgres: > writer process > 20190 postgres 13 -2 3219m 71m 68m S 0 0.9 0:52.48 postgres: > cribq cribq [local] idle > > I am writing moderately large (~3k) records to my database a few times > a second. Even when I stop doing that, the process continues to take > up all of that memory. > > Am I reading this right? Why is it using so much memory? shared_buffers? -- Alvaro Herrera http://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc.
On 8/12/09 9:44 PM, "Alex" <alex@liivid.com> wrote: > The writer process seems to be using inordinate amounts of memory: > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ > COMMAND > 11088 postgres 13 -2 3217m 2.9g 2.9g S 0 38.7 0:10.46 postgres: > writer process > 20190 postgres 13 -2 3219m 71m 68m S 0 0.9 0:52.48 postgres: > cribq cribq [local] idle > > I am writing moderately large (~3k) records to my database a few times > a second. Even when I stop doing that, the process continues to take > up all of that memory. > > Am I reading this right? Why is it using so much memory? > It is exclusively using the difference between the RES and SHR columns. So ... Less than ~50MB and likely much less than that (2.9g - 2.9g with rounding error). SHR is the shared memory the process has touched, and is shared amongst all postgres processes. Typically, this maxes out at the value of your shared_buffers setting. Based on the above, I'd wager your shared_buffers setting is 3000MB. If your writer process or any other process has a value for (RES - SHR) that is very large, then be concerned. For example, the second postgres process in the above top output is using about 3MB exclusively, but has touched about 68MB of the shared space, and so it shows up as 68 + 3 = 71m in the RES column. 3MB is not much so this is not a concern. > > -- > Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) > To make changes to your subscription: > http://www.postgresql.org/mailpref/pgsql-performance >
I am confused about what the OS is reporting for memory usage on CentOS 5.3 Linux. Looking at the resident memory size of the processes. Looking at the resident size of all postgres processes, the system should be using around 30Gb of physical ram. I know that it states that it is using a lot of shared memory. My question is how to I determine how much physical RAM postgres is using at any point in time?
This server has 24Gb of ram, and is reporting that 23GB is free for use. See calculation below
(Memory Total – Used) + (Buffers + Cached) = Free Memory
(24675740 – 24105052) + (140312 + 22825616) = 23,536,616 or ~23 Gigabytes
So if my server has 23Gb of ram that is free for use, why is postgres reporting resident sizes of 30GB? Shared memory is reporting the same values, so how is the OS reporting that only 1Gb of RAM is being used?
Help?
top - 12:43:41 up 2 days, 19:04, 2 users, load average: 4.99, 4.81, 4.33
Tasks: 245 total, 4 running, 241 sleeping, 0 stopped, 0 zombie
Cpu(s): 26.0%us, 0.0%sy, 0.0%ni, 73.9%id, 0.1%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 24675740k total, 24105052k used, 570688k free, 140312k buffers
Swap: 2097144k total, 272k used, 2096872k free, 22825616k cached
---------------------
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
19469 postgres 15 0 8324m 7.9g 7.9g S 0.0 33.7 0:54.30 postgres: writer process
29763 postgres 25 0 8329m 4.5g 4.5g R 99.8 19.0 24:53.02 postgres: niadmin database x.x.x.49(51136) UPDATE
29765 postgres 25 0 8329m 4.4g 4.4g R 99.8 18.8 24:42.77 postgres: niadmin database x.x.x.49(51138) UPDATE
31778 postgres 25 0 8329m 4.2g 4.2g R 99.5 17.8 17:56.95 postgres: niadmin database x.x.x.49(51288) UPDATE
31779 postgres 25 0 8329m 4.2g 4.2g R 99.1 17.8 17:59.62 postgres: niadmin database x.x.x.49(51289) UPDATE
31780 postgres 23 0 8329m 4.1g 4.1g R 100.1 17.5 17:52.53 postgres: niadmin database x.x.x.49(51290) UPDATE
19467 postgres 15 0 8320m 160m 160m S 0.0 0.7 0:00.24 /opt/PostgreSQL/8.3/bin/postgres -D /opt/PostgreSQL/8.3/data
19470 postgres 15 0 8324m 2392 1880 S 0.0 0.0 0:01.72 postgres: wal writer process
This server has 24Gb of ram, and is reporting that 23GB is free for use. See calculation below
(Memory Total – Used) + (Buffers + Cached) = Free Memory
(24675740 – 24105052) + (140312 + 22825616) = 23,536,616 or ~23 Gigabytes
So if my server has 23Gb of ram that is free for use, why is postgres reporting resident sizes of 30GB? Shared memory is reporting the same values, so how is the OS reporting that only 1Gb of RAM is being used?
Help?
top - 12:43:41 up 2 days, 19:04, 2 users, load average: 4.99, 4.81, 4.33
Tasks: 245 total, 4 running, 241 sleeping, 0 stopped, 0 zombie
Cpu(s): 26.0%us, 0.0%sy, 0.0%ni, 73.9%id, 0.1%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 24675740k total, 24105052k used, 570688k free, 140312k buffers
Swap: 2097144k total, 272k used, 2096872k free, 22825616k cached
---------------------
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
19469 postgres 15 0 8324m 7.9g 7.9g S 0.0 33.7 0:54.30 postgres: writer process
29763 postgres 25 0 8329m 4.5g 4.5g R 99.8 19.0 24:53.02 postgres: niadmin database x.x.x.49(51136) UPDATE
29765 postgres 25 0 8329m 4.4g 4.4g R 99.8 18.8 24:42.77 postgres: niadmin database x.x.x.49(51138) UPDATE
31778 postgres 25 0 8329m 4.2g 4.2g R 99.5 17.8 17:56.95 postgres: niadmin database x.x.x.49(51288) UPDATE
31779 postgres 25 0 8329m 4.2g 4.2g R 99.1 17.8 17:59.62 postgres: niadmin database x.x.x.49(51289) UPDATE
31780 postgres 23 0 8329m 4.1g 4.1g R 100.1 17.5 17:52.53 postgres: niadmin database x.x.x.49(51290) UPDATE
19467 postgres 15 0 8320m 160m 160m S 0.0 0.7 0:00.24 /opt/PostgreSQL/8.3/bin/postgres -D /opt/PostgreSQL/8.3/data
19470 postgres 15 0 8324m 2392 1880 S 0.0 0.0 0:01.72 postgres: wal writer process
On Fri, 2009-08-14 at 14:00 -0400, Jeremy Carroll wrote: > I am confused about what the OS is reporting for memory usage on > CentOS 5.3 Linux. Looking at the resident memory size of the > processes. Looking at the resident size of all postgres processes, the > system should be using around 30Gb of physical ram. I know that it > states that it is using a lot of shared memory. My question is how to > I determine how much physical RAM postgres is using at any point in > time? > > This server has 24Gb of ram, and is reporting that 23GB is free for > use. See calculation below > > (Memory Total – Used) + (Buffers + Cached) = Free Memory > (24675740 – 24105052) + (140312 + 22825616) = 23,536,616 or ~23 > Gigabytes > you're using cached swap in your calculation ( 22825616 ) swap is not RAM -- it's disk > > So if my server has 23Gb of ram that is free for use, why is postgres > reporting resident sizes of 30GB? Shared memory is reporting the same > values, so how is the OS reporting that only 1Gb of RAM is being used? you have 570688k free RAM + 140312k buffers RAM This looks to me like the OS is saying that you are using 24105052k used > > Help? > > top - 12:43:41 up 2 days, 19:04, 2 users, load average: 4.99, 4.81, > 4.33 > Tasks: 245 total, 4 running, 241 sleeping, 0 stopped, 0 zombie > Cpu(s): 26.0%us, 0.0%sy, 0.0%ni, 73.9%id, 0.1%wa, 0.0%hi, 0.0% > si, 0.0%st > Mem: 24675740k total, 24105052k used, 570688k free, 140312k > buffers > Swap: 2097144k total, 272k used, 2096872k free, 22825616k > cached > --------------------- > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ > COMMAND > 19469 postgres 15 0 8324m 7.9g 7.9g S 0.0 33.7 0:54.30 postgres: > writer process > > 29763 postgres 25 0 8329m 4.5g 4.5g R 99.8 19.0 24:53.02 postgres: > niadmin database x.x.x.49(51136) UPDATE > 29765 postgres 25 0 8329m 4.4g 4.4g R 99.8 18.8 24:42.77 postgres: > niadmin database x.x.x.49(51138) UPDATE > 31778 postgres 25 0 8329m 4.2g 4.2g R 99.5 17.8 17:56.95 postgres: > niadmin database x.x.x.49(51288) UPDATE > 31779 postgres 25 0 8329m 4.2g 4.2g R 99.1 17.8 17:59.62 postgres: > niadmin database x.x.x.49(51289) UPDATE > 31780 postgres 23 0 8329m 4.1g 4.1g R 100.1 17.5 17:52.53 > postgres: niadmin database x.x.x.49(51290) UPDATE > 19467 postgres 15 0 8320m 160m 160m S 0.0 0.7 > 0:00.24 /opt/PostgreSQL/8.3/bin/postgres > -D /opt/PostgreSQL/8.3/data > 19470 postgres 15 0 8324m 2392 1880 S 0.0 0.0 0:01.72 postgres: > wal writer process
But the kernel can take back any of the cache memory if it wants to. Therefore it is free memory.
This still does not explain why the top command is reporting ~9GB of resident memory, yet the top command does not suggest that any physical memory is being used.
On 8/14/09 2:43 PM, "Reid Thompson" <reid.thompson@ateb.com> wrote:
This still does not explain why the top command is reporting ~9GB of resident memory, yet the top command does not suggest that any physical memory is being used.
On 8/14/09 2:43 PM, "Reid Thompson" <reid.thompson@ateb.com> wrote:
you're using cached swap in your calculation ( 22825616 ) swap is not
RAM -- it's disk
On Fri, Aug 14, 2009 at 12:00 PM, Jeremy Carroll<jeremy.carroll@networkedinsights.com> wrote: > I am confused about what the OS is reporting for memory usage on CentOS 5.3 > Linux. Looking at the resident memory size of the processes. Looking at the > resident size of all postgres processes, the system should be using around > 30Gb of physical ram. I know that it states that it is using a lot of shared > memory. My question is how to I determine how much physical RAM postgres is > using at any point in time? OK, take the first pg process, and write down its RES size. For all the rest, write down RES-SHR for how much more it's using. Since they use a lot of shared memory, and since you're showing something like 7.9G shared, I'm gonna guess that's the size of your shared_buffers. With those numbers you should get something just over a shade of 7.9G used, and most of that is shared_buffers. Also, a quick check is to look at this number: 22825616k cached which tells you how much memory the OS is using for cache, which is ~22G. I note that you've got 2G swapped out, this might well be shared_buffers or something you'd rather not have swapped out. Look into setting your swappiness lower (5 or so should do) to stop the OS from swapping so much out. /sbin/sysctl -a|grep swappiness vm.swappiness = 60 is the default. You can change it permanently by editing your /etc/sysctl.conf file (or wherever it lives) and rebooting, or running /sbin/sysctl -p to process the entries and make them stick this session. My big servers run with swappiness of 1 with no problems. > > This server has 24Gb of ram, and is reporting that 23GB is free for use. See > calculation below > > (Memory Total – Used) + (Buffers + Cached) = Free Memory > (24675740 – 24105052) + (140312 + 22825616) = 23,536,616 or ~23 Gigabytes > > > So if my server has 23Gb of ram that is free for use, why is postgres > reporting resident sizes of 30GB? Shared memory is reporting the same > values, so how is the OS reporting that only 1Gb of RAM is being used? > > Help? > > top - 12:43:41 up 2 days, 19:04, 2 users, load average: 4.99, 4.81, 4.33 > Tasks: 245 total, 4 running, 241 sleeping, 0 stopped, 0 zombie > Cpu(s): 26.0%us, 0.0%sy, 0.0%ni, 73.9%id, 0.1%wa, 0.0%hi, 0.0%si, > 0.0%st > Mem: 24675740k total, 24105052k used, 570688k free, 140312k buffers > Swap: 2097144k total, 272k used, 2096872k free, 22825616k cached > --------------------- > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 19469 postgres 15 0 8324m 7.9g 7.9g S 0.0 33.7 0:54.30 postgres: > writer process > 29763 postgres 25 0 8329m 4.5g 4.5g R 99.8 19.0 24:53.02 postgres: > niadmin database x.x.x.49(51136) UPDATE > 29765 postgres 25 0 8329m 4.4g 4.4g R 99.8 18.8 24:42.77 postgres: > niadmin database x.x.x.49(51138) UPDATE > 31778 postgres 25 0 8329m 4.2g 4.2g R 99.5 17.8 17:56.95 postgres: > niadmin database x.x.x.49(51288) UPDATE > 31779 postgres 25 0 8329m 4.2g 4.2g R 99.1 17.8 17:59.62 postgres: > niadmin database x.x.x.49(51289) UPDATE > 31780 postgres 23 0 8329m 4.1g 4.1g R 100.1 17.5 17:52.53 postgres: > niadmin database x.x.x.49(51290) UPDATE > 19467 postgres 15 0 8320m 160m 160m S 0.0 0.7 0:00.24 > /opt/PostgreSQL/8.3/bin/postgres -D /opt/PostgreSQL/8.3/data > > 19470 postgres 15 0 8324m 2392 1880 S 0.0 0.0 0:01.72 postgres: wal > writer process -- When fascism comes to America, it will be intolerance sold as diversity.
I'm betting it's shared_buffers that have been swapped out (2G swapped out on his machine) for kernel cache. The RES and SHR being the same says the actual processes are using hardly any ram, just hitting shared_buffers. On Fri, Aug 14, 2009 at 2:20 PM, Jeremy Carroll<jeremy.carroll@networkedinsights.com> wrote: > But the kernel can take back any of the cache memory if it wants to. > Therefore it is free memory. > > This still does not explain why the top command is reporting ~9GB of > resident memory, yet the top command does not suggest that any physical > memory is being used. > > > On 8/14/09 2:43 PM, "Reid Thompson" <reid.thompson@ateb.com> wrote: > > you're using cached swap in your calculation ( 22825616 ) swap is not > RAM -- it's disk > -- When fascism comes to America, it will be intolerance sold as diversity.
On 8/14/09 11:00 AM, "Jeremy Carroll" <jeremy.carroll@networkedinsights.com> wrote: > I am confused about what the OS is reporting for memory usage on CentOS 5.3 > Linux. Looking at the resident memory size of the processes. Looking at the > resident size of all postgres processes, the system should be using around > 30Gb of physical ram. I know that it states that it is using a lot of shared > memory. My question is how to I determine how much physical RAM postgres is > using at any point in time? Resident includes Shared. Shared is shared. So you have to subtract it from all the processes to see what they use on their own. What you really want is RES-SHR, or some of the other columns available in top. Hit 'h' in top to get some help on the other columns available, and 'f' and 'o' manipulate them. In particular, you might find the "DATA" column useful. It is approximately RES-SHR-CODE > > This server has 24Gb of ram, and is reporting that 23GB is free for use. See > calculation below > > (Memory Total Used) + (Buffers + Cached) = Free Memory > (24675740 24105052) + (140312 + 22825616) = 23,536,616 or ~23 Gigabytes > > > So if my server has 23Gb of ram that is free for use, why is postgres > reporting resident sizes of 30GB? Shared memory is reporting the same values, > so how is the OS reporting that only 1Gb of RAM is being used? > > Help? > > top - 12:43:41 up 2 days, 19:04, 2 users, load average: 4.99, 4.81, 4.33 > Tasks: 245 total, 4 running, 241 sleeping, 0 stopped, 0 zombie > Cpu(s): 26.0%us, 0.0%sy, 0.0%ni, 73.9%id, 0.1%wa, 0.0%hi, 0.0%si, 0.0%st > Mem: 24675740k total, 24105052k used, 570688k free, 140312k buffers > Swap: 2097144k total, 272k used, 2096872k free, 22825616k cached > --------------------- > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 19469 postgres 15 0 8324m 7.9g 7.9g S 0.0 33.7 0:54.30 postgres: writer > process > 29763 postgres 25 0 8329m 4.5g 4.5g R 99.8 19.0 24:53.02 postgres: niadmin > database x.x.x.49(51136) UPDATE Lets just take the two above and pretend that they are the only postgres processes. The RAM used by each exclusively is RES-SHR. Or, close to nothing for these two, aside from the rounding error. The memory used by postgres for shared memory is the largest of all SHR columns for postgres columns. Or, about 7.9GB. So, postgres is using about 7.9GB for shared memory, and very little for anything else. In formula form, its close to SUM(RES) - SUM(SHR) + MAX(SHR). That doesn't cover everything, but is very close. See the other columns available in top.
This is postgres 8.4 BTW. It says 2.9Gb of RESIDENT memory, that also seems to be shared. Is this the writer sharing the records it wrote in a shared buffer? PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 11088 postgres 13 -2 3217m 3.0g 3.0g S 0 39.5 0:14.23 postgres: writer process 968 postgres 14 -2 3219m 1.4g 1.4g S 0 18.8 4:37.57 postgres: cribq cribq [local] idle 24593 postgres 13 -2 3219m 331m 327m S 0 4.3 0:10.12 postgres: cribq cribq [local] idle 26181 postgres 13 -2 3219m 323m 319m S 0 4.2 0:06.48 postgres: cribq cribq [local] idle 12504 postgres 14 -2 3219m 297m 293m S 0 3.9 0:02.71 postgres: cribq cribq [local] idle 13565 postgres 14 -2 3219m 292m 288m S 0 3.8 0:02.75 postgres: cribq cribq [local] idle 623 postgres 13 -2 3219m 292m 287m S 0 3.8 0:02.28 postgres: cribq cribq [local] idle On Thu, Aug 13, 2009 at 1:29 PM, Alvaro Herrera<alvherre@commandprompt.com> wrote: > Alex wrote: >> The writer process seems to be using inordinate amounts of memory: >> >> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ >> COMMAND >> 11088 postgres 13 -2 3217m 2.9g 2.9g S 0 38.7 0:10.46 postgres: >> writer process >> 20190 postgres 13 -2 3219m 71m 68m S 0 0.9 0:52.48 postgres: >> cribq cribq [local] idle >> >> I am writing moderately large (~3k) records to my database a few times >> a second. Even when I stop doing that, the process continues to take >> up all of that memory. >> >> Am I reading this right? Why is it using so much memory? > > shared_buffers? > > -- > Alvaro Herrera http://www.CommandPrompt.com/ > The PostgreSQL Company - Command Prompt, Inc. > -- Alex Neth Liivid, Inc www.liivid.com +1 206 499 4995 +86 13761577188 Stephen Leacock - "I detest life-insurance agents: they always argue that I shall some day die, which is not so." - http://www.brainyquote.com/quotes/authors/s/stephen_leacock.html
If I have 10GB of ram, and I see a process using 5Gb of RES size. Then TOP should at least report 5GB physical memory
used(AKA: Not available in CACHED, or FREE). If I run a 'free -m', I should only see 5GB of ram available. I can
understandwith virtual memory that some of it may be on disk, therefore I may not see this memory being taken away from
thephysical memory available.
I am thoroughly confused that TOP is reporting that I have 99% of my physical RAM free, while the process list suggests
thatsome are taking ~8Gb of Resident (Physical) Memory. Any explanation as to why TOP is reporting this? I have a
PostgreSQL8.3 server with 48Gb of RAM on a Dell R610 server that is reporting that 46.5GB of RAM is free. This confuses
meto no end. Why is it not reporting much more physical memory used?
[root@pg6 jcarroll]# free -m
total used free shared buffers cached
Mem: 48275 48136 138 0 141 46159
-/+ buffers/cache: 1835 46439
Swap: 2047 12 2035
Thanks!
top - 09:24:38 up 17:05, 1 user, load average: 1.09, 1.08, 1.18
Tasks: 239 total, 2 running, 237 sleeping, 0 stopped, 0 zombie
Cpu(s): 6.2%us, 0.1%sy, 0.0%ni, 93.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 49433916k total, 49295460k used, 138456k free, 145308k buffers
Swap: 2097144k total, 12840k used, 2084304k free, 47267056k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
13200 postgres 15 0 16.1g 15g 15g R 2.3 33.6 2:36.78 postgres: writer process
29029 postgres 25 0 16.1g 13g 13g R 99.9 29.4 36:35.13 postgres: dbuser database 192.168.200.8(36979) UPDATE
13198 postgres 15 0 16.1g 317m 316m S 0.0 0.7 0:00.57 /opt/PostgreSQL/8.3/bin/postgres -D
/opt/PostgreSQL/8.3/data
13201 postgres 15 0 16.1g 2300 1824 S 0.0 0.0 0:00.39 postgres: wal writer process
13202 postgres 15 0 98.7m 1580 672 S 0.0 0.0 0:15.12 postgres: stats collector process
-----Original Message-----
From: Scott Carey [mailto:scott@richrelevance.com]
Sent: Friday, August 14, 2009 3:38 PM
To: Jeremy Carroll; pgsql-performance@postgresql.org
Subject: Re: [PERFORM] Memory reporting on CentOS Linux
On 8/14/09 11:00 AM, "Jeremy Carroll" <jeremy.carroll@networkedinsights.com>
wrote:
> I am confused about what the OS is reporting for memory usage on CentOS 5.3
> Linux. Looking at the resident memory size of the processes. Looking at the
> resident size of all postgres processes, the system should be using around
> 30Gb of physical ram. I know that it states that it is using a lot of shared
> memory. My question is how to I determine how much physical RAM postgres is
> using at any point in time?
Resident includes Shared. Shared is shared. So you have to subtract it
from all the processes to see what they use on their own. What you really
want is RES-SHR, or some of the other columns available in top. Hit 'h' in
top to get some help on the other columns available, and 'f' and 'o'
manipulate them. In particular, you might find the "DATA" column useful.
It is approximately RES-SHR-CODE
>
> This server has 24Gb of ram, and is reporting that 23GB is free for use. See
> calculation below
>
> (Memory Total Used) + (Buffers + Cached) = Free Memory
> (24675740 24105052) + (140312 + 22825616) = 23,536,616 or ~23 Gigabytes
>
>
> So if my server has 23Gb of ram that is free for use, why is postgres
> reporting resident sizes of 30GB? Shared memory is reporting the same values,
> so how is the OS reporting that only 1Gb of RAM is being used?
>
> Help?
>
> top - 12:43:41 up 2 days, 19:04, 2 users, load average: 4.99, 4.81, 4.33
> Tasks: 245 total, 4 running, 241 sleeping, 0 stopped, 0 zombie
> Cpu(s): 26.0%us, 0.0%sy, 0.0%ni, 73.9%id, 0.1%wa, 0.0%hi, 0.0%si, 0.0%st
> Mem: 24675740k total, 24105052k used, 570688k free, 140312k buffers
> Swap: 2097144k total, 272k used, 2096872k free, 22825616k cached
> ---------------------
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 19469 postgres 15 0 8324m 7.9g 7.9g S 0.0 33.7 0:54.30 postgres: writer
> process
> 29763 postgres 25 0 8329m 4.5g 4.5g R 99.8 19.0 24:53.02 postgres: dbuser
> database x.x.x.49(51136) UPDATE
Lets just take the two above and pretend that they are the only postgres
processes.
The RAM used by each exclusively is RES-SHR. Or, close to nothing for these
two, aside from the rounding error.
The memory used by postgres for shared memory is the largest of all SHR
columns for postgres columns. Or, about 7.9GB. So, postgres is using
about 7.9GB for shared memory, and very little for anything else.
In formula form, its close to
SUM(RES) - SUM(SHR) + MAX(SHR).
That doesn't cover everything, but is very close. See the other columns
available in top.
Jeremy Carroll <jeremy.carroll@networkedinsights.com> writes:
> I am thoroughly confused that TOP is reporting that I have 99% of my
> physical RAM free, while the process list suggests that some are
> taking ~8Gb of Resident (Physical) Memory. Any explanation as to why
> TOP is reporting this? I have a PostgreSQL 8.3 server with 48Gb of RAM
> on a Dell R610 server that is reporting that 46.5GB of RAM is free.
Exactly where do you draw that conclusion from? I see "free 138M".
It does look like there's something funny about top's accounting for
shared memory --- maybe it's counting it as "cached"? It's hardly
unusual for top to give bogus numbers in the presence of shared memory,
of course, but this seems odd :-(. With such large amounts of RAM
involved I wonder if there could be an overflow problem. You might file
a bug against top in whatever distro you are using.
regards, tom lane
Linux strives to always use 100% of memory at any given time. Therefore the system will always throw free memory into
swapcache. The kernel will (and can) take any memory away from the swap cache at any time for resident (physical)
memoryfor processes.
That's why they have the column "-/+ buffers/cache:". That shows 46Gb Free RAM.
I cannot be the only person that has asked this question.
-----Original Message-----
From: Tom Lane [mailto:tgl@sss.pgh.pa.us]
Sent: Saturday, August 15, 2009 10:25 AM
To: Jeremy Carroll
Cc: Scott Carey; pgsql-performance@postgresql.org
Subject: Re: [PERFORM] Memory reporting on CentOS Linux
Jeremy Carroll <jeremy.carroll@networkedinsights.com> writes:
> I am thoroughly confused that TOP is reporting that I have 99% of my
> physical RAM free, while the process list suggests that some are
> taking ~8Gb of Resident (Physical) Memory. Any explanation as to why
> TOP is reporting this? I have a PostgreSQL 8.3 server with 48Gb of RAM
> on a Dell R610 server that is reporting that 46.5GB of RAM is free.
Exactly where do you draw that conclusion from? I see "free 138M".
It does look like there's something funny about top's accounting for
shared memory --- maybe it's counting it as "cached"? It's hardly
unusual for top to give bogus numbers in the presence of shared memory,
of course, but this seems odd :-(. With such large amounts of RAM
involved I wonder if there could be an overflow problem. You might file
a bug against top in whatever distro you are using.
regards, tom lane
On 08/15/2009 11:39 AM, Jeremy Carroll wrote: > Linux strives to always use 100% of memory at any given time. Therefore the system will always throw free memory into swapcache. The kernel will (and can) take any memory away from the swap cache at any time for resident (physical) memoryfor processes. > > That's why they have the column "-/+ buffers/cache:". That shows 46Gb Free RAM. > > I cannot be the only person that has asked this question. > I vote for screwed up reporting over some PostgreSQL-specific explanation. My understanding of RSS is the same as you suggested earlier - if 50% RAM is listed as resident, then there should not be 90%+ RAM free. I cannot think of anything PostgreSQL might be doing into influencing this to be false. Do you get the same results after reboot? :-) I'm serious. Memory can be corrupted, and Linux can have bugs. I would not think that cache memory shows up as RSS for a particular process. Cache memory is shared by the entire system, and is not allocated towards any specific process. Or, at least, this is my understanding. Just for kicks, I tried an mmap() scenario (I do not think PostgreSQL uses mmap()), and it showed a large RSS, but it did NOT show free memory. Cheers, mark > -----Original Message----- > From: Tom Lane [mailto:tgl@sss.pgh.pa.us] > Sent: Saturday, August 15, 2009 10:25 AM > To: Jeremy Carroll > Cc: Scott Carey; pgsql-performance@postgresql.org > Subject: Re: [PERFORM] Memory reporting on CentOS Linux > > Jeremy Carroll<jeremy.carroll@networkedinsights.com> writes: > >> I am thoroughly confused that TOP is reporting that I have 99% of my >> physical RAM free, while the process list suggests that some are >> taking ~8Gb of Resident (Physical) Memory. Any explanation as to why >> TOP is reporting this? I have a PostgreSQL 8.3 server with 48Gb of RAM >> on a Dell R610 server that is reporting that 46.5GB of RAM is free. >> > Exactly where do you draw that conclusion from? I see "free 138M". > > It does look like there's something funny about top's accounting for > shared memory --- maybe it's counting it as "cached"? It's hardly > unusual for top to give bogus numbers in the presence of shared memory, > of course, but this seems odd :-(. With such large amounts of RAM > involved I wonder if there could be an overflow problem. You might file > a bug against top in whatever distro you are using. > > regards, tom lane > > -- Mark Mielke<mark@mielke.cc>
On Fri, Aug 14, 2009 at 3:43 PM, Reid Thompson<reid.thompson@ateb.com> wrote: > On Fri, 2009-08-14 at 14:00 -0400, Jeremy Carroll wrote: >> I am confused about what the OS is reporting for memory usage on >> CentOS 5.3 Linux. Looking at the resident memory size of the >> processes. Looking at the resident size of all postgres processes, the >> system should be using around 30Gb of physical ram. I know that it >> states that it is using a lot of shared memory. My question is how to >> I determine how much physical RAM postgres is using at any point in >> time? >> >> This server has 24Gb of ram, and is reporting that 23GB is free for >> use. See calculation below >> >> (Memory Total – Used) + (Buffers + Cached) = Free Memory >> (24675740 – 24105052) + (140312 + 22825616) = 23,536,616 or ~23 >> Gigabytes >> > you're using cached swap in your calculation ( 22825616 ) swap is not > RAM -- it's disk As far as I know, cached is only on the next line as a formatting convenience. It is unrelated to swap and not on disk. What could "cached swap" possibly mean anyway? Having said that, as a Linux user of many years, I have found that per-process memory reporting is completely worthless. All you can really do is look at the overall statistics for the box and try to get some sense as to whether the box, over all, is struggling. Trying to understand how individual processes are contributing to that is an inexact science when it's not a complete waste of time. ...Robert
On Sat, 15 Aug 2009, Mark Mielke wrote: > I vote for screwed up reporting over some PostgreSQL-specific explanation. My > understanding of RSS is the same as you suggested earlier - if 50% RAM is > listed as resident, then there should not be 90%+ RAM free. I cannot think of > anything PostgreSQL might be doing into influencing this to be false. The only thing I would have thought that would allow this would be mmap. > Just for kicks, I tried an mmap() scenario (I do not think PostgreSQL uses > mmap()), and it showed a large RSS, but it did NOT show free memory. More details please. What did you do, and what happened? I would have thought that a large read-only mmapped file that has been read (and therefore is in RAM) would be counted as VIRT and RES of the process in top, but can clearly be evicted from the cache at any time, and therefore would show up as buffer or cache rather than process memory in the totals. +1 on the idea that Linux memory reporting is incomprehensible nowadays. Matthew -- There once was a limerick .sig that really was not very big It was going quite fine Till it reached the fourth line
I believe this is exactly what is happening. I see that the TOP output lists a large amount ov VIRT & RES size being used, but the kernel does not report this memory as being reserved and instead lists it as free memory or cached.
If this is indeed the case, how does one determine if a PostgreSQL instance requires more memory? Or how to determine if the system is using memory efficiently?
Thanks for the responses.
On 8/17/09 6:03 AM, "Matthew Wakeling" <matthew@flymine.org> wrote:
If this is indeed the case, how does one determine if a PostgreSQL instance requires more memory? Or how to determine if the system is using memory efficiently?
Thanks for the responses.
On 8/17/09 6:03 AM, "Matthew Wakeling" <matthew@flymine.org> wrote:
On Sat, 15 Aug 2009, Mark Mielke wrote:
> I vote for screwed up reporting over some PostgreSQL-specific explanation. My
> understanding of RSS is the same as you suggested earlier - if 50% RAM is
> listed as resident, then there should not be 90%+ RAM free. I cannot think of
> anything PostgreSQL might be doing into influencing this to be false.
The only thing I would have thought that would allow this would be mmap.
> Just for kicks, I tried an mmap() scenario (I do not think PostgreSQL uses
> mmap()), and it showed a large RSS, but it did NOT show free memory.
More details please. What did you do, and what happened? I would have
thought that a large read-only mmapped file that has been read (and
therefore is in RAM) would be counted as VIRT and RES of the process in
top, but can clearly be evicted from the cache at any time, and therefore
would show up as buffer or cache rather than process memory in the totals.
+1 on the idea that Linux memory reporting is incomprehensible nowadays.
Matthew
--
There once was a limerick .sig
that really was not very big
It was going quite fine
Till it reached the fourth line
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
On 8/17/09 10:24 AM, "Jeremy Carroll" <jeremy.carroll@networkedinsights.com> wrote: > I believe this is exactly what is happening. I see that the TOP output lists a > large amount ov VIRT & RES size being used, but the kernel does not report > this memory as being reserved and instead lists it as free memory or cached. Oh! I recall I found that fun behaviour Linux and thought it was a Postgres bug a while back. It has lot of other bad effects on how the kernel chooses to swap. I really should have recalled that one. Due to this behavior, I had initially blamed postgres for "pinning" memory in shared_buffers in the disk cache. But that symptom is one of linux thinking somehow that pages read into shared memory are still cached (or something similar). Basically, it thinks that there is more free memory than there is when there is a lot of shared memory. Run a postgres instance with over 50% memory assigned to shared_buffers and when memory pressure builds kswapd will go NUTS in CPU use, apparently confused. With high OS 'swappiness' value it will swap in and out too much, and with low 'swappiness' it will CPU spin, aware on one hand that it is low on memory, but confused by the large apparent amount free so it doesn't free up much and kswapd chews up all the CPU and the system almost hangs. It behaves as if the logic that determines where to get memory from for a process knows that its almost out, but the logic that decides what to swap out thinks that there is plenty free. The larger the ratio of shared memory to total memory in the system, the higher the CPU use by the kernel when managing the buffer cache. Bottom line is that Linux plus lots of SYSV shared mem doesn't work as well as it should. Setting shared_buffers past 35% RAM doesn't work well on Linux. Shared memory accounting is fundamentally broken in Linux (see some other threads on how the OOM killer works WRT shared memory for other examples). > > If this is indeed the case, how does one determine if a PostgreSQL instance > requires more memory? Or how to determine if the system is using memory > efficiently? Just be aware that the definite memory used per process is RES-SHR, and that the max SHR value is mostly duplicated in the 'cached' or 'free' columns. That mas SHR value IS used by postgres, and not the OS cache. If cached + memory free is on the order of your shared_buffers/SHR size, you're pretty much out of memory. Additionally, the OS will start putting things into swap before you reach that point, so pay attention to the swap used column in top or free. That is a more reliable indicator than anything else at the system level. If you want to know what postgres process is using the most memory on its own look at the DATA and CODE top columns, or calculate RES-SHR. I have no idea if more recent Linux Kernels have fixed this at all. > > Thanks for the responses. > > > On 8/17/09 6:03 AM, "Matthew Wakeling" <matthew@flymine.org> wrote: > >> On Sat, 15 Aug 2009, Mark Mielke wrote: >>> I vote for screwed up reporting over some PostgreSQL-specific explanation. >>> My >>> understanding of RSS is the same as you suggested earlier - if 50% RAM is >>> listed as resident, then there should not be 90%+ RAM free. I cannot think >>> of >>> anything PostgreSQL might be doing into influencing this to be false. >> >> The only thing I would have thought that would allow this would be mmap. >> >>> Just for kicks, I tried an mmap() scenario (I do not think PostgreSQL uses >>> mmap()), and it showed a large RSS, but it did NOT show free memory. >> >> More details please. What did you do, and what happened? I would have >> thought that a large read-only mmapped file that has been read (and >> therefore is in RAM) would be counted as VIRT and RES of the process in >> top, but can clearly be evicted from the cache at any time, and therefore >> would show up as buffer or cache rather than process memory in the totals. >> >> +1 on the idea that Linux memory reporting is incomprehensible nowadays. >> >> Matthew >> >> -- >> There once was a limerick .sig >> that really was not very big >> It was going quite fine >> Till it reached the fourth line >> >> -- >> Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) >> To make changes to your subscription: >> http://www.postgresql.org/mailpref/pgsql-performance >> >
On 8/17/09 4:43 PM, "Scott Carey" <scott@richrelevance.com> wrote: > > > On 8/17/09 10:24 AM, "Jeremy Carroll" <jeremy.carroll@networkedinsights.com> > wrote: > >> I believe this is exactly what is happening. I see that the TOP output lists >> a >> large amount ov VIRT & RES size being used, but the kernel does not report >> this memory as being reserved and instead lists it as free memory or cached. > > Oh! I recall I found that fun behaviour Linux and thought it was a Postgres > bug a while back. It has lot of other bad effects on how the kernel chooses > to swap. I really should have recalled that one. Due to this behavior, I > had initially blamed postgres for "pinning" memory in shared_buffers in the > disk cache. But that symptom is one of linux thinking somehow that pages > read into shared memory are still cached (or something similar). > > Basically, it thinks that there is more free memory than there is when there > is a lot of shared memory. Run a postgres instance with over 50% memory > assigned to shared_buffers and when memory pressure builds kswapd will go > NUTS in CPU use, apparently confused. With high OS 'swappiness' value it > will swap in and out too much, and with low 'swappiness' it will CPU spin, > aware on one hand that it is low on memory, but confused by the large > apparent amount free so it doesn't free up much and kswapd chews up all the > CPU and the system almost hangs. It behaves as if the logic that determines > where to get memory from for a process knows that its almost out, but the > logic that decides what to swap out thinks that there is plenty free. The > larger the ratio of shared memory to total memory in the system, the higher > the CPU use by the kernel when managing the buffer cache. > Based on a little digging, I'd say that this patch to the kernel probably alleviates the performance problems I've seen with swapping when shared mem is high: http://lwn.net/Articles/286472/ Other patches have improved the shared memory tracking, but its not clear if tools like top have taken advantage of the new info available in /proc.
On Fri, 14 Aug 2009, Scott Carey wrote: > The memory used by postgres for shared memory is the largest of all SHR > columns for postgres columns. Or, about 7.9GB. So, postgres is using > about 7.9GB for shared memory, and very little for anything else. It's a good idea to check this result against the actual shared memory block allocated. If the server has been up long enough to go through all of shared_buffers once, the results should be close. You can look at the block under Linux using "ipcs -m"; the one you want should look something like this: ------ Shared Memory Segments -------- key shmid owner perms bytes nattch status 0x0052e2c1 21757972 gsmith 600 548610048 10 That represents a bit over 512MB worth of allocated memory for the server. Alternately, you can use "pmap -d" on a PostgreSQL process to find the block, something like this works: $ pmap -d 13961 | egrep "^Address|shmid" Address Kbytes Mode Offset Device Mapping 96c41000 535752 rw-s- 0000000000000000 000:00009 [ shmid=0x14c0014 ] I have given up on presuming the summary values top shows are good for anything on Linux. I look at /proc/meminfo to see how much RAM is free, and to figure out what's going on with the server processes I use: ps -e -o pid,rss,vsz,size,cmd | grep postgres And compute my own totals (one of these days I'm going to script that process). Useful reading on this topic: http://virtualthreads.blogspot.com/2006/02/understanding-memory-usage-on-linux.html http://mail.nl.linux.org/linux-mm/2003-03/msg00077.html http://forums.gentoo.org/viewtopic.php?t=175419 Most confusion about what's going on here can be resolved by spending some quality time with pmap. -- * Greg Smith gsmith@gregsmith.com http://www.gregsmith.com Baltimore, MD