On Wed, Mar 26, 2014 at 08:22:01PM +0000, Anand Kumar, Karthik wrote:
> Looking a little deeper, I saw signs of memory being heavily fragmented:
>
> root@site-db01b:/var/log # cat /proc/buddyinfo
> Node 0, zone DMA 1 1 2 2 2 1 0 0 1 1 3
> Node 0, zone DMA32 8 7 8 7 10 8 7 11 9 5 92
> Node 0, zone Normal 13069 0 0 0 0 0 0 0 0 0 1
> Node 1, zone Normal 652315 36885 1168 0 1 1 0 1 1 1 0
>
>
> Node 0 has 13069 4k blocks, and zero 8k blocks available to use
> Which is likely what caused the problem, I'd think.
>
> A little while later though, buddyinfo changed and suddenly there was a
> lot more memory in 8k blocks.
>
> root@site-db01b:/proc # cat /proc/buddyinfo
> Node 0, zone DMA 1 1 2 2 2 1 0
> 0 1 1 3
> Node 0, zone DMA32 8 7 8 7 10 8 7
> 11 9 5 92
> Node 0, zone Normal 9645 5495 1115 0 0 0 0
> 0 0 0 1
> Node 1, zone Normal 409734 10953 1 0 1 1 0
> 1 1 1 0
>
> (Note the change in the Node 0 line, 8k blocks went up from 0 to 5495)
>
> Anyone have any idea why memory was so fragmented, and what causes memory
> to be defragged? Is it something postgres does? Are there any kernel
> specific settings that control it?
If I had to take a guess, it is zone_reclaim; see:
http://frosty-postgres.blogspot.com/2012/08/postgresql-numa-and-zone-reclaim-mode.html
The fix is this sysctl:
vm.zone_reclaim_mode = 0
--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com
+ Everyone has their own god. +
