Thread: high shared buffer and swap
Friendly greetings ! I found something "odd" (something that i can't explain) this weekend. An octocore server with 32GB of ram, running postgresql 8.3.6 Running only postgresql, slony-I and pgbouncer. Just for testing purpose, i tried a setting with 26GB of shared_buffer. I quickly noticed that the performances wasn't very good and the server started to swap slowly but surely. (but still up to 2000query/second as reported by pgfouine) It used all the 2GB of swap. I removed the server from production, added 10GB of swap and left it for the weekend with only slony and postgresql up to keep it in sync with the master database. This morning i found that the whole 12GB of swap were used : Mem: 32892008k total, 32714728k used, 177280k free, 70872k buffers Swap: 12582896k total, 12531812k used, 51084k free, 27047696k cached # cat /proc/meminfo MemTotal: 32892008 kB MemFree: 171140 kB Buffers: 70852 kB Cached: 27065208 kB SwapCached: 4752492 kB Active: 24362168 kB Inactive: 7806884 kB HighTotal: 0 kB HighFree: 0 kB LowTotal: 32892008 kB LowFree: 171140 kB SwapTotal: 12582896 kB SwapFree: 53064 kB Dirty: 122636 kB Writeback: 0 kB AnonPages: 280336 kB Mapped: 14118588 kB Slab: 224632 kB PageTables: 235120 kB NFS_Unstable: 0 kB Bounce: 0 kB CommitLimit: 29028900 kB Committed_AS: 28730620 kB VmallocTotal: 34359738367 kB VmallocUsed: 12916 kB VmallocChunk: 34359725307 kB While i understand that a very high shared_buffer wasn't a good idea, i don't understand this behaviour. Any tought ? I tried this setup because having 2 level of data caching doesn't make sense to me. (1 in OS filesystem cache and 1 in shm (shared_buffer)). I'd love to understand what's happening here ! Thank you :) -- F4FQM Kerunix Flan Laurent Laborde
Sorry for top-posting - the iphone mail client sucks. I think what's happening is that the sytem is seeing that some pages of shared memory haven't been used recently and because there's more shared memory than filesystem cache less recently than the filesystem cache pages. So it pages out the shared memory. This is really awful because we use a kind of lru algorithm for shared memory so the pages that it's paging out are precisely the pges likely to be used soon. I wonder if we should try to mlock shared buffers. -- Greg On 4 May 2009, at 10:10, Laurent Laborde <kerdezixe@gmail.com> wrote: > Friendly greetings ! > I found something "odd" (something that i can't explain) this weekend. > > An octocore server with 32GB of ram, running postgresql 8.3.6 > Running only postgresql, slony-I and pgbouncer. > > Just for testing purpose, i tried a setting with 26GB of > shared_buffer. > > I quickly noticed that the performances wasn't very good and the > server started to swap slowly but surely. > (but still up to 2000query/second as reported by pgfouine) > > It used all the 2GB of swap. > I removed the server from production, added 10GB of swap and left it > for the weekend with only slony and postgresql up to keep it in sync > with the master database. > > This morning i found that the whole 12GB of swap were used : > Mem: 32892008k total, 32714728k used, 177280k free, 70872k > buffers > Swap: 12582896k total, 12531812k used, 51084k free, 27047696k > cached > > # cat /proc/meminfo > MemTotal: 32892008 kB > MemFree: 171140 kB > Buffers: 70852 kB > Cached: 27065208 kB > SwapCached: 4752492 kB > Active: 24362168 kB > Inactive: 7806884 kB > HighTotal: 0 kB > HighFree: 0 kB > LowTotal: 32892008 kB > LowFree: 171140 kB > SwapTotal: 12582896 kB > SwapFree: 53064 kB > Dirty: 122636 kB > Writeback: 0 kB > AnonPages: 280336 kB > Mapped: 14118588 kB > Slab: 224632 kB > PageTables: 235120 kB > NFS_Unstable: 0 kB > Bounce: 0 kB > CommitLimit: 29028900 kB > Committed_AS: 28730620 kB > VmallocTotal: 34359738367 kB > VmallocUsed: 12916 kB > VmallocChunk: 34359725307 kB > > While i understand that a very high shared_buffer wasn't a good idea, > i don't understand this behaviour. > Any tought ? > > I tried this setup because having 2 level of data caching doesn't make > sense to me. (1 in OS filesystem cache and 1 in shm (shared_buffer)). > > I'd love to understand what's happening here ! Thank you :) > > -- > F4FQM > Kerunix Flan > Laurent Laborde > > -- > Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org > ) > To make changes to your subscription: > http://www.postgresql.org/mailpref/pgsql-performance
On Mon, May 04, 2009 at 10:57:47AM +0200, Greg Stark wrote: > I think what's happening is that the sytem is seeing that some pages of > shared memory haven't been used recently and because there's more shared > memory than filesystem cache less recently than the filesystem cache > pages. So it pages out the shared memory. This is really awful because we > use a kind of lru algorithm for shared memory so the pages that it's > paging out are precisely the pges likely to be used soon. > > I wonder if we should try to mlock shared buffers. You can try, but it probably won't work. You often need to be root to lock pages, and even on Linux 2.6.9+ where you don't need to be root there's a limit of 32KB (that's only my machine anyway). Sure, that can be changed, if you're root. Actually locking the shared buffers seems to me like a footgun. People occasionally give postgresql masses of memory leaving not enough to run the rest of the system. Locking the memory would make the situation worse. Personally I've never seen a benefit of setting shared buffer above the expected working set size. I generally let the kernel share the remaining memory between postgresql disk cache and other processes I might be running. On a NUMA machine you want to be keeping your memory on the local node and letting the kernel copy that data from elsewhere to your local memory when you need it. Have a nice day, -- Martijn van Oosterhout <kleptog@svana.org> http://svana.org/kleptog/ > Please line up in a tree and maintain the heap invariant while > boarding. Thank you for flying nlogn airlines.
Attachment
On Mon, May 4, 2009 at 2:10 AM, Laurent Laborde <kerdezixe@gmail.com> wrote: > Friendly greetings ! > I found something "odd" (something that i can't explain) this weekend. > > An octocore server with 32GB of ram, running postgresql 8.3.6 > Running only postgresql, slony-I and pgbouncer. > > Just for testing purpose, i tried a setting with 26GB of shared_buffer. > > I quickly noticed that the performances wasn't very good and the > server started to swap slowly but surely. > (but still up to 2000query/second as reported by pgfouine) > > It used all the 2GB of swap. > I removed the server from production, added 10GB of swap and left it > for the weekend with only slony and postgresql up to keep it in sync > with the master database. > > This morning i found that the whole 12GB of swap were used : > Mem: 32892008k total, 32714728k used, 177280k free, 70872k buffers > Swap: 12582896k total, 12531812k used, 51084k free, 27047696k cached Try setting swappiness =0. But as someone else mentioned, I've alwas had better luck letting the OS do most of the caching anyway.
> An octocore server with 32GB of ram, running postgresql 8.3.6
> Running only postgresql, slony-I and pgbouncer.
>
> Just for testing purpose, i tried a setting with 26GB of shared_buffer.
>
> I quickly noticed that the performances wasn't very good and the
> server started to swap slowly but surely.
> (but still up to 2000query/second as reported by pgfouine)
>
> It used all the 2GB of swap.
> I removed the server from production, added 10GB of swap and left it
> for the weekend with only slony and postgresql up to keep it in sync
> with the master database.
>
> This morning i found that the whole 12GB of swap were used :
Hm, do you really need swap with 32Gb of RAM ?
One could argue "yes but swap is useful to avoid out of memory errors".
But if a loaded server starts to swap a lot, it is as good as dead
anyway...
On Tue, May 5, 2009 at 11:15 AM, PFC <lists@peufeu.com> wrote: > >> An octocore server with 32GB of ram, running postgresql 8.3.6 >> Running only postgresql, slony-I and pgbouncer. >> >> Just for testing purpose, i tried a setting with 26GB of shared_buffer. >> >> I quickly noticed that the performances wasn't very good and the >> server started to swap slowly but surely. >> (but still up to 2000query/second as reported by pgfouine) >> >> It used all the 2GB of swap. >> I removed the server from production, added 10GB of swap and left it >> for the weekend with only slony and postgresql up to keep it in sync >> with the master database. >> >> This morning i found that the whole 12GB of swap were used : > > Hm, do you really need swap with 32Gb of RAM ? > One could argue "yes but swap is useful to avoid out of memory > errors". > But if a loaded server starts to swap a lot, it is as good as dead > anyway... Not really, but we have it. I tried with swappinness set to 0 and ... it swaps ! I'm back to 4GB of shared_buffer :) I'll try various setting, maybe 16GB, etc ... But my goal was to avoid OS filesystem cache and usage of shared_buffer instead : FAIL. -- F4FQM Kerunix Flan Laurent Laborde
On Tue, 5 May 2009, Laurent Laborde wrote: > I tried with swappinness set to 0 and ... it swaps ! While I wouldn't presume to try and teach you to suck eggs, once you set swappiness to zero the system will take a while to settle down. The swappiness setting will stop it swapping out, but if there is stuff out in swap, then it won't force it to swap it in early. So the machine will still thrash when it tries to access something that is out on swap. Easy way to solve this is to swapoff the various swap partitions, which will force it all into memory. You can swapon them again afterwards. Matthew -- Surely the value of C++ is zero, but C's value is now 1? -- map36, commenting on the "No, C++ isn't equal to D. 'C' is undeclared [...] C++ should really be called 1" response to "C++ -- shouldn't it be called D?"