Thread: NFS, file system cache and shared_buffers
I just learned that NFS does not use a file system cache on the client side. On the other hand, PostgreSQL relies on the file system cache for performance, because beyond a certain amount of shared_buffers performance will suffer. Together these things seem to indicate that you cannot get good performance with a large database over NFS since you can leverage memory speed. Now I wonder if there are any remedies (CacheFS?) and what experiences people have made with the performance of large databases over NFS. Yours, Laurenz Albe
On 05/27/2014 02:06 PM, Albe Laurenz wrote: > I just learned that NFS does not use a file system cache on the client side. > > On the other hand, PostgreSQL relies on the file system cache for performance, > because beyond a certain amount of shared_buffers performance will suffer. > > Together these things seem to indicate that you cannot get good performance > with a large database over NFS since you can leverage memory speed. > > Now I wonder if there are any remedies (CacheFS?) and what experiences > people have made with the performance of large databases over NFS. I have no personal experience with NFS, but sounds like a higher-than-usual shared_buffers value would be good. - Heikki
* Heikki Linnakangas (hlinnakangas@vmware.com) wrote:
> On 05/27/2014 02:06 PM, Albe Laurenz wrote:
> >I just learned that NFS does not use a file system cache on the client side.
> >
> >On the other hand, PostgreSQL relies on the file system cache for performance,
> >because beyond a certain amount of shared_buffers performance will suffer.
> >
> >Together these things seem to indicate that you cannot get good performance
> >with a large database over NFS since you can leverage memory speed.
> >
> >Now I wonder if there are any remedies (CacheFS?) and what experiences
> >people have made with the performance of large databases over NFS.
>
> I have no personal experience with NFS, but sounds like a
> higher-than-usual shared_buffers value would be good.
It would certainly be worthwhile to test it. In the end you would,
hopefully, end up with a situation where you're maximizing RAM usage-
the NFS server is certainly caching in *its* filesystem cache, while on
the PG server you're getting the benefit of shared_buffers without the
drawback of double-buffering (since you couldn't ever use the NFS
server's memory for shared_buffers anyway).
All that said, there has always been a recommendation of caution around
using NFS as a backing store for PG, or any RDBMS..
Thanks,
Stephen
Attachment
Stephen Frost wrote: > All that said, there has always been a recommendation of caution around > using NFS as a backing store for PG, or any RDBMS.. I know that Oracle recommends it - they even built an NFS client into their database server to make the most of it. Yours, Laurenz Albe
On 05/27/2014 10:00 AM, Albe Laurenz wrote: > I know that Oracle recommends it - they even built an NFS client > into their database server to make the most of it. That's odd. Every time the subject of NFS comes up, it's almost immediately shot down with explicit advice to Never Do That(tm). It can be kinda safe-ish if mounted in sync mode with caching disabled, but I'd never use it on any of our systems. We also have this in the Wiki: http://wiki.postgresql.org/wiki/Shared_Storage -- Shaun Thomas OptionsHouse, LLC | 141 W. Jackson Blvd. | Suite 800 | Chicago IL, 60604 312-676-8870 sthomas@optionshouse.com ______________________________________________ See http://www.peak6.com/email_disclaimer/ for terms and conditions related to this email
On 5/27/2014 9:09 AM, Shaun Thomas wrote: > On 05/27/2014 10:00 AM, Albe Laurenz wrote: > >> I know that Oracle recommends it - they even built an NFS client >> into their database server to make the most of it. > > That's odd. Every time the subject of NFS comes up, it's almost > immediately shot down with explicit advice to Never Do That(tm). It > can be kinda safe-ish if mounted in sync mode with caching disabled, > but I'd never use it on any of our systems. It has been a long time since I was in the weeds of this issue, but the crux is that it was (still is?) hard to be sure that the filesystem's behavior was exactly as expected. My recollection of the Oracle story was that they had to verify the end-to-end behavior, and essentially certify its correctness to guarantee database acid. So you needed to be running a very specific version of the NFS code, configured in a very specific way. This isn't entirely inconsistent with the reference above that they "built an NFS client". That's something you might need to do in order to be sure it behaves in the way you expect. Possibly the NFS implementations deployed today are more consistent and correct than was the case in the past. I wouldn't use a network filesystem for any kind of database storage myself though.
On Tue, May 27, 2014 at 8:00 AM, Albe Laurenz <laurenz.albe@wien.gv.at> wrote:
Stephen Frost wrote:I know that Oracle recommends it - they even built an NFS client
> All that said, there has always been a recommendation of caution around
> using NFS as a backing store for PG, or any RDBMS..
into their database server to make the most of it.
Last I heard (which has been a while), Oracle supported specific brand named implementations of NFS, and warned against any others on a data integrity basis.
Why would they implement their own client? Did they have to do something special in their client to make it safe?
Cheers,
Jeff
On Tue, May 27, 2014 at 4:06 AM, Albe Laurenz <laurenz.albe@wien.gv.at> wrote:
I just learned that NFS does not use a file system cache on the client side.
My experience suggested that it did something a little weirder than that. It would cache read data as long as it was clean, but once the data was dirtied and written back, it would drop it from the cache. But it probably depends on lot of variables and details I don't recall anymore.
On the other hand, PostgreSQL relies on the file system cache for performance,
because beyond a certain amount of shared_buffers performance will suffer.
Some people have some problems sometimes, and they are not readily reproducible (at least not in a publicly disclosable way, that I know of). Other people use large shared_buffers and have no problems at all, or none that are fixed by lowering shared_buffers.
We should not elevate a rumor to a law.
Together these things seem to indicate that you cannot get good performance
with a large database over NFS since you can leverage memory speed.
Now I wonder if there are any remedies (CacheFS?) and what experiences
people have made with the performance of large databases over NFS.
I've only used it in cases where I didn't consider durability important, and even then didn't find it worth pursuing due to the performance. But I was piggybacking on existing resource, I didn't have an impressive NFS server tuned specifically for this usage, so my experience probably doesn't mean much performance wise.
Cheers,
Jeff
John Melesky wrote: >> I just learned that NFS does not use a file system cache on the client side. > > That's ... incorrect. NFS is cache-capable. NFSv3 (I think? It may have been v2) started sending > metadata on file operations that was intended to allow for client-side caches. NFSv4 added all sorts > of stateful behavior which allows for much more aggressive caching. What do you mean by "allows"? Does it cache files in memory or not? Do you need additional software? Special configuration? > Where did you read that you could not use caching with NFS? I have it by hearsay from somebody who seemed knowledgable, and the existence of CacheFS seemed to indicate it was true. Yours, Laurenz Albe
Jeff Janes wrote: >>> All that said, there has always been a recommendation of caution around >>> using NFS as a backing store for PG, or any RDBMS.. >> >> I know that Oracle recommends it - they even built an NFS client >> into their database server to make the most of it. > > Last I heard (which has been a while), Oracle supported specific brand named implementations of NFS, > and warned against any others on a data integrity basis. I couldn't find any detailed information, but it seems that only certain NFS devices are supported. > Why would they implement their own client? Did they have to do something special in their client to > make it safe? I think it is mostly a performance issue. Each backend mounts its own copy of the data files it needs. Yours, Laurenz Albe
On Tue, May 27, 2014 at 4:06 AM, Albe Laurenz <laurenz.albe@wien.gv.at> wrote:
I just learned that NFS does not use a file system cache on the client side.
That's ... incorrect. NFS is cache-capable. NFSv3 (I think? It may have been v2) started sending metadata on file operations that was intended to allow for client-side caches. NFSv4 added all sorts of stateful behavior which allows for much more aggressive caching.
Where did you read that you could not use caching with NFS?
John Melesky | Sr Database Administrator
503.284.7581 x204 | john.melesky@rentrak.com
RENTRAK | www.rentrak.com | NASDAQ: RENT
Notice: This message is confidential and is intended only for the
recipient(s) named above. If you have received this message in error,
or are not the named recipient(s), please immediately notify the
sender and delete this message.
RENTRAK | www.rentrak.com | NASDAQ: RENT
Notice: This message is confidential and is intended only for the
recipient(s) named above. If you have received this message in error,
or are not the named recipient(s), please immediately notify the
sender and delete this message.
>> Why would they implement their own client? Did they have to do something special in their client to >> make it safe? > > I think it is mostly a performance issue. Each backend mounts its own copy > of the data files it needs. I personally would never put PostgreSQL on an NFS share on Linux. Unless things have changed radically in the last couple years, Linux's NFS code is flaky and unreliable, including flat-out lying about whether stuff has been sent and received or not. This is why NetApp's NFS servers came with their own, proprietary, Linux kernel module. NFS on Solaris/Illumos is a different story. Not sure about FreeBSD. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com
I wrote: >Jeff Janes wrote: >>>> All that said, there has always been a recommendation of caution around >>>> using NFS as a backing store for PG, or any RDBMS.. >>> >>> I know that Oracle recommends it - they even built an NFS client >>> into their database server to make the most of it. >> >> Last I heard (which has been a while), Oracle supported specific brand named implementations of NFS, >> and warned against any others on a data integrity basis. > > I couldn't find any detailed information, but it seems that only certain > NFS devices are supported. For the record: Oracle support told me that all NFS is supported on Linux, regardless of the device. "Supported" does not mean "recommended", of course. Yours, Laurenz Albe