Re: Linux machine aggressively clearing cache - Mailing list pgsql-performance

From Dave Crooke
Subject Re: Linux machine aggressively clearing cache
Date
Msg-id CALi4UpjQSQ3FeShXsLRkG8p5cyuRpJLzy7_hmLr0DRsZ_qVK+A@mail.gmail.com
Whole thread Raw
In response to Linux machine aggressively clearing cache  (Joshua Berkus <josh@agliodbs.com>)
Responses Re: Linux machine aggressively clearing cache  (Joshua Berkus <josh@agliodbs.com>)
List pgsql-performance
This may just be a typo, but if you really did create write (dirty) block device cache by writing the pg_dump file somewhere, then that is what it's supposed to do ;) Linux is more aggressive about write cache and will allow more of it to build up than e.g. HP-UX which will start to throttle process-to-cache writes to avoid getting too far behind.

Read cache of course does not need to be flushed and can simply be dumped when the memory is needed, and so Linux will keep more or less unlimited amounts of read cache until it needs the memory for something else .... here is an output from "free" on my laptop, showing ~2.5GB of read cache that can be freed almost instantly if needed for process memory, write cache, kernel buffers, etc. The -/+ line shows a net of what is being used by processes.

dave:~$ free
             total       used       free     shared    buffers     cached
Mem:       8089056    7476424     612632          0     603508    2556584
-/+ buffers/cache:    4316332    3772724
Swap:     24563344    1176284   23387060

redirecting  pg_dump >/dev/null  will read the DB without writing anything, but it's pretty resource intensive .... if you just want to get the database tables into the OS read cache you can do it much more cheaply with   sudo tar cvf - /var/lib/postgresql/8.4/main/base | cat >/dev/null  or similar (GNU tar somehow detects if you connect its stdout directly to /dev/null and then it cheats and doesn't do the reads)

In the second "free" output below, the kernel has grabbed what it can for cache, leaving only ~64MB of actual free memory for instant use.

dave:~$ pg_dump -F c hyper9db >/dev/null
dave:~$ free
             total       used       free     shared    buffers     cached
Mem:       8089056    8024252      64804          0     287432    3797956
-/+ buffers/cache:    3938864    4150192
Swap:     24563344    1166556   23396788
dave:~$

Cheers
Dave

On Tue, Mar 27, 2012 at 3:06 PM, Joshua Berkus <josh@agliodbs.com> wrote:
... but Linux rapidly cleared the cache (flushing to disk) down to 25GB within an hour.

--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com
San Francisco

--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance

pgsql-performance by date:

Previous
From: Claudio Freire
Date:
Subject: Re: Linux machine aggressively clearing cache
Next
From: Francois Deliege
Date:
Subject: Re: Distinct + Limit