Thread: FreeBSD config

FreeBSD config

From
Dror Matalon
Date:
Hi,

We have postgres running on freebsd 4.9 with 2 Gigs of memory. As per
repeated advice on the mailing lists we configured effective_cache_size
= 25520 which you get by doing `sysctl -n vfs.hibufspace` / 8192

Which results in using 200Megs for disk caching.

Is there a reason not to increase the hibufspace beyond the 200 megs and
provide a bigger cache to postgres? I looked both on the postgres and
freebsd mailing lists and couldn't find a good answer to this.

If yes, any suggestions on what would be a good size on a 2 Gig machine?

Regards,

Dror


--
Dror Matalon
Zapatec Inc
1700 MLK Way
Berkeley, CA 94709
http://www.fastbuzz.com
http://www.zapatec.com

Re: FreeBSD config

From
"scott.marlowe"
Date:
On Thu, 26 Feb 2004, Dror Matalon wrote:

> Hi,
>
> We have postgres running on freebsd 4.9 with 2 Gigs of memory. As per
> repeated advice on the mailing lists we configured effective_cache_size
> = 25520 which you get by doing `sysctl -n vfs.hibufspace` / 8192
>
> Which results in using 200Megs for disk caching.
>
> Is there a reason not to increase the hibufspace beyond the 200 megs and
> provide a bigger cache to postgres? I looked both on the postgres and
> freebsd mailing lists and couldn't find a good answer to this.

Actually, I think you're confusing effective_cache_size with
shared_buffers.

effective_cache_size changes no cache settings for postgresql, it simply
acts as a hint to the planner on about how much of the dataset your OS /
Kernel / Disk cache can hold.

Making it bigger only tells the query planny it's more likely the data
it's looking for will be in cache.

shared_buffers, OTOH, sets the amount of cache that postgresql uses.  It's
generall considered that 256 Megs or 1/4 of memory, whichever is LESS, is
a good setting for production database servers.


Re: FreeBSD config

From
Dror Matalon
Date:
On Thu, Feb 26, 2004 at 11:55:31AM -0700, scott.marlowe wrote:
> On Thu, 26 Feb 2004, Dror Matalon wrote:
>
> > Hi,
> >
> > We have postgres running on freebsd 4.9 with 2 Gigs of memory. As per
> > repeated advice on the mailing lists we configured effective_cache_size
> > = 25520 which you get by doing `sysctl -n vfs.hibufspace` / 8192
> >
> > Which results in using 200Megs for disk caching.
> >
> > Is there a reason not to increase the hibufspace beyond the 200 megs and
> > provide a bigger cache to postgres? I looked both on the postgres and
> > freebsd mailing lists and couldn't find a good answer to this.
>
> Actually, I think you're confusing effective_cache_size with
> shared_buffers.

No, I'm not.

>
> effective_cache_size changes no cache settings for postgresql, it simply
> acts as a hint to the planner on about how much of the dataset your OS /
> Kernel / Disk cache can hold.

I understand that. The question is why have the OS, in this case FreeBsd
use only 200 Megs for disk cache and not more. Why not double the
vfs.hibufspace  to 418119680 and double the effective_cache_size to 51040.

>
> Making it bigger only tells the query planny it's more likely the data
> it's looking for will be in cache.
>
> shared_buffers, OTOH, sets the amount of cache that postgresql uses.  It's
> generall considered that 256 Megs or 1/4 of memory, whichever is LESS, is
> a good setting for production database servers.
>

Actually last I looked, I thought that the recommended max shared
buffers was 10,000, 80MB,  even on machines with large amounts of memory.

Regards,

Dror

--
Dror Matalon
Zapatec Inc
1700 MLK Way
Berkeley, CA 94709
http://www.fastbuzz.com
http://www.zapatec.com

Re: FreeBSD config

From
"scott.marlowe"
Date:
On Thu, 26 Feb 2004, Dror Matalon wrote:

> On Thu, Feb 26, 2004 at 11:55:31AM -0700, scott.marlowe wrote:
> > On Thu, 26 Feb 2004, Dror Matalon wrote:
> >
> > > Hi,
> > >
> > > We have postgres running on freebsd 4.9 with 2 Gigs of memory. As per
> > > repeated advice on the mailing lists we configured effective_cache_size
> > > = 25520 which you get by doing `sysctl -n vfs.hibufspace` / 8192
> > >
> > > Which results in using 200Megs for disk caching.
> > >
> > > Is there a reason not to increase the hibufspace beyond the 200 megs and
> > > provide a bigger cache to postgres? I looked both on the postgres and
> > > freebsd mailing lists and couldn't find a good answer to this.
> >
> > Actually, I think you're confusing effective_cache_size with
> > shared_buffers.
>
> No, I'm not.

OK, sorry, I wasn't sure which you meant.

> > effective_cache_size changes no cache settings for postgresql, it simply
> > acts as a hint to the planner on about how much of the dataset your OS /
> > Kernel / Disk cache can hold.
>
> I understand that. The question is why have the OS, in this case FreeBsd
> use only 200 Megs for disk cache and not more. Why not double the
> vfs.hibufspace  to 418119680 and double the effective_cache_size to 51040.

Doesn't the kernel just use the spare memory to buffer anyway?

I'd say if you got 2 megs memory and nothing else on the box, give a big
chunk (1 gig or so) to the kernel to manage.  Unless large kernel caches
cause some issues in FreeBSD.

> > Making it bigger only tells the query planny it's more likely the data
> > it's looking for will be in cache.
> >
> > shared_buffers, OTOH, sets the amount of cache that postgresql uses.  It's
> > generall considered that 256 Megs or 1/4 of memory, whichever is LESS, is
> > a good setting for production database servers.
> >
>
> Actually last I looked, I thought that the recommended max shared
> buffers was 10,000, 80MB,  even on machines with large amounts of memory.

It really depends on what you're doing.  For loads involving very large
data sets, up to 256 Megs has resulted in improvements, but anything after
that has only had advantages in very limited types of applications.


Re: FreeBSD config

From
Christopher Kings-Lynne
Date:
> We have postgres running on freebsd 4.9 with 2 Gigs of memory. As per
> repeated advice on the mailing lists we configured effective_cache_size
> = 25520 which you get by doing `sysctl -n vfs.hibufspace` / 8192
>
> Which results in using 200Megs for disk caching.

effective_cache_size does nothing of the sort.  CHeck your
shared_buffers value...

> Is there a reason not to increase the hibufspace beyond the 200 megs and
> provide a bigger cache to postgres? I looked both on the postgres and
> freebsd mailing lists and couldn't find a good answer to this.

Well, maybe butnot necessarily.  It's better to leave the OS to look after
most of your RAM.

Chris


Re: FreeBSD config

From
Dror Matalon
Date:
On Fri, Feb 27, 2004 at 05:47:47AM +0800, Christopher Kings-Lynne wrote:
> > We have postgres running on freebsd 4.9 with 2 Gigs of memory. As per
> > repeated advice on the mailing lists we configured effective_cache_size
> > = 25520 which you get by doing `sysctl -n vfs.hibufspace` / 8192
> >
> > Which results in using 200Megs for disk caching.
>
> effective_cache_size does nothing of the sort.  CHeck your
> shared_buffers value...

Sigh.

http://www.varlena.com/varlena/GeneralBits/Tidbits/annotated_conf_e.html

effective_cache_size
    Sets the optimizer's assumption about the effective size of the disk
    cache (that is, the portion of the kernel's disk cache that will be
    used for PostgreSQL data files). This is measured in disk pages, which
    are normally 8 kB each.


http://archives.postgresql.org/pgsql-performance/2003-07/msg00159.php
talks about how to programmatically determine the right setting for
effective_cache_size:
    case `uname` in "FreeBSD")
    echo "effective_cache_size = $((`sysctl -n vfs.hibufspace` / 8192))"
                    ;;
                    *)
    echo "Unable to automatically determine the effective cache size" >> /dev/stderr
                ;;
                esac

which brings me back to my question why not make Freebsd use more of its
memory for disk caching and then tell postgres about it.



>
> > Is there a reason not to increase the hibufspace beyond the 200 megs and
> > provide a bigger cache to postgres? I looked both on the postgres and
> > freebsd mailing lists and couldn't find a good answer to this.
>
> Well, maybe butnot necessarily.  It's better to leave the OS to look after
> most of your RAM.
>
> Chris
>

--
Dror Matalon
Zapatec Inc
1700 MLK Way
Berkeley, CA 94709
http://www.fastbuzz.com
http://www.zapatec.com

Re: FreeBSD config

From
"Kevin Barnard"
Date:
On 26 Feb 2004 at 13:58, Dror Matalon wrote:

>
> which brings me back to my question why not make Freebsd use more of its
> memory for disk caching and then tell postgres about it.
>

I think there is some confusion about maxbufsize and hibufspace.  I looking at a comment in the FreeBSB  source 4.9 that explains this.  I think you will want to increase effective_cache to match maxbufsize not hibufspace but I could be wrong.

$FreeBSD: src/sys/kern/vfs_bio.c,v 1.242.2.21 line 363

Re: FreeBSD config

From
Dror Matalon
Date:
Thanks for the pointer. So

maxbufspace = nbuf * BKVASIZE;

Which is confirmed in
http://unix.derkeiler.com/Mailing-Lists/FreeBSD/performance/2003-09/0045.html

and it looks like there's a patch by Sean Chittenden at
http://people.freebsd.org/~seanc/patches/patch-HEAD-kern.nbuf

that does what I was asking. Seems a little on the bleeding edge. Has
anyone tried this?


On Thu, Feb 26, 2004 at 04:36:01PM -0600, Kevin Barnard wrote:
> On 26 Feb 2004 at 13:58, Dror Matalon wrote:
>
> >
> > which brings me back to my question why not make Freebsd use more of its
> > memory for disk caching and then tell postgres about it.
> >
>
> I think there is some confusion about maxbufsize and hibufspace.  I looking at a
> comment in the FreeBSB  source 4.9 that explains this.  I think you will want to
> increase effective_cache to match maxbufsize not hibufspace but I could be wrong.
>
> $FreeBSD: src/sys/kern/vfs_bio.c,v 1.242.2.21 line 363
>

--
Dror Matalon
Zapatec Inc
1700 MLK Way
Berkeley, CA 94709
http://www.fastbuzz.com
http://www.zapatec.com

Re: FreeBSD config

From
Dror Matalon
Date:
On Thu, Feb 26, 2004 at 06:06:06PM -0500, Christopher Weimann wrote:
> On 02/26/2004-11:16AM, Dror Matalon wrote:
> > >
> > > effective_cache_size changes no cache settings for postgresql, it simply
> > > acts as a hint to the planner on about how much of the dataset your OS /
> > > Kernel / Disk cache can hold.
> >
> > I understand that. The question is why have the OS, in this case FreeBsd
> > use only 200 Megs for disk cache and not more. Why not double the
> > vfs.hibufspace  to 418119680 and double the effective_cache_size to 51040.
> >
>
> FreeBSD uses ALL ram that isn't being used for something else as
> its disk cache.  The "effective_cache_size" in the PostGreSQL config
> has no effect on how the OS chooses to use memory, it is just hint
> to the PostGreSQL planner so it can guess the the likelyhood of
> what it is looking for being in the cache.

Let me try and say it again. I know that setting effective_cache_size
doesn't affect the OS' cache. I know it just gives Postgres the *idea*
of how much cache the OS is using. I know that. I also know that a
correct hint helps performance.

I've read Matt Dillon's discussion about the freebsd VM at
http://www.daemonnews.org/200001/freebsd_vm.html and I didn't see him
saying that Freebsd uses all the free RAM for disk cache. Would you care
to provide a URL pointing to that?

Assuming you are correct, why has the ongoing recommendation been to use
hibufspace/8192 as the effective_cache_size? Seems like it would be
quite a bit more on machines with lots of RAM.

Regards,

Dror

--
Dror Matalon
Zapatec Inc
1700 MLK Way
Berkeley, CA 94709
http://www.fastbuzz.com
http://www.zapatec.com

Re: FreeBSD config

From
Shridhar Daithankar
Date:
Dror Matalon wrote:

> Let me try and say it again. I know that setting effective_cache_size
> doesn't affect the OS' cache. I know it just gives Postgres the *idea*
> of how much cache the OS is using. I know that. I also know that a
> correct hint helps performance.
>
> I've read Matt Dillon's discussion about the freebsd VM at
> http://www.daemonnews.org/200001/freebsd_vm.html and I didn't see him
> saying that Freebsd uses all the free RAM for disk cache. Would you care
> to provide a URL pointing to that?

I don't believe freeBSD yses everything available unlike linux. It is actually a
good thing. If you have 1GB RAM and kernel buffers set at 600MB, you are
guaranteed to have some mmory in crunch situations.

As far you original questions, I think you can increase the kernel buffer sizes
for VFS safely. However remembet that more to dedicate to kernel buffers, less
space you have in case of crunch for whatever reasons.

FreeBSD gives you a control which linux does not. Use it to best of your advantage..

  Shridhar

Re: FreeBSD config

From
"scott.marlowe"
Date:
On Fri, 27 Feb 2004, Shridhar Daithankar wrote:

> Dror Matalon wrote:
>
> > Let me try and say it again. I know that setting effective_cache_size
> > doesn't affect the OS' cache. I know it just gives Postgres the *idea*
> > of how much cache the OS is using. I know that. I also know that a
> > correct hint helps performance.
> >
> > I've read Matt Dillon's discussion about the freebsd VM at
> > http://www.daemonnews.org/200001/freebsd_vm.html and I didn't see him
> > saying that Freebsd uses all the free RAM for disk cache. Would you care
> > to provide a URL pointing to that?
>
> I don't believe freeBSD yses everything available unlike linux. It is actually a
> good thing. If you have 1GB RAM and kernel buffers set at 600MB, you are
> guaranteed to have some mmory in crunch situations.

Linux doesn't work with a pre-assigned size for kernel cache.
It just grabs whatever's free, minus a few megs for easily launching new
programs or allocating more memory for programs, and uses that for the
cache.  then, when a request comes in for more memory than is free, it
dumps some of the least used buffers and gives them back.

It would seem to work very well underneath a mixed load server like an
LAPP box.


Re: FreeBSD config

From
Dror Matalon
Date:
I guess the thing to do is to move this topic over to a freebsd list
where we can get more definitive answers on how disk caching is handled.
I asked here since I know that FreeBsd is often recommended,
http://www.varlena.com/varlena/GeneralBits/Tidbits/perf.html#
as a good platform for postgres, and with Modern machines often having
Gigabytes of memory the issue of, possibly, having a disk cache of 200MB
would be one often asked.

On Fri, Feb 27, 2004 at 12:46:08PM +0530, Shridhar Daithankar wrote:
> Dror Matalon wrote:
>
> >Let me try and say it again. I know that setting effective_cache_size
> >doesn't affect the OS' cache. I know it just gives Postgres the *idea*
> >of how much cache the OS is using. I know that. I also know that a
> >correct hint helps performance.
> >
> >I've read Matt Dillon's discussion about the freebsd VM at
> >http://www.daemonnews.org/200001/freebsd_vm.html and I didn't see him
> >saying that Freebsd uses all the free RAM for disk cache. Would you care
> >to provide a URL pointing to that?
>
> I don't believe freeBSD yses everything available unlike linux. It is
> actually a good thing. If you have 1GB RAM and kernel buffers set at 600MB,
> you are guaranteed to have some mmory in crunch situations.
>
> As far you original questions, I think you can increase the kernel buffer
> sizes for VFS safely. However remembet that more to dedicate to kernel
> buffers, less space you have in case of crunch for whatever reasons.
>
> FreeBSD gives you a control which linux does not. Use it to best of your
> advantage..
>
>  Shridhar

--
Dror Matalon
Zapatec Inc
1700 MLK Way
Berkeley, CA 94709
http://www.fastbuzz.com
http://www.zapatec.com

Re: FreeBSD config

From
Vivek Khera
Date:
>>>>> "DM" == Dror Matalon <dror@zapatec.com> writes:

DM> which brings me back to my question why not make Freebsd use more of its
DM> memory for disk caching and then tell postgres about it.

Because this is a painfully hard thing to do ;-(

It involves hacking a system header file and recompiling the kernel.
It is not a simple tunable.  It has side effects regarding some other
sizing parameter as well, but I don't recall the details.  Details
have been posted to this list at least once before by Sean Chittenden.

--
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Vivek Khera, Ph.D.                Khera Communications, Inc.
Internet: khera@kciLink.com       Rockville, MD  +1-301-869-4449 x806
AIM: vivekkhera Y!: vivek_khera   http://www.khera.org/~vivek/

Re: FreeBSD config

From
Shridhar Daithankar
Date:
On Friday 27 February 2004 21:03, scott.marlowe wrote:
> Linux doesn't work with a pre-assigned size for kernel cache.
> It just grabs whatever's free, minus a few megs for easily launching new
> programs or allocating more memory for programs, and uses that for the
> cache.  then, when a request comes in for more memory than is free, it
> dumps some of the least used buffers and gives them back.
>
> It would seem to work very well underneath a mixed load server like an
> LAPP box.

I was just pointing out that freeBSD is different than linux nd for one thing
it is good because if there is a bug in freeSD VM, it won't run rampant
because you can explicitly limit kernel cache and other parameter.

OTOH, freeBSD VM anyways works. And running unlimited kernel cache allowed
linux to iron out some of corner cases bugs.

Not a concern anymore I believe but having choice is always great..

Shridhar

Re: FreeBSD config

From
Christopher Weimann
Date:
On 02/26/2004-01:58PM, Dror Matalon wrote:
>
> Sigh.
>

Sigh, right back at you.

> which brings me back to my question why not make Freebsd use more of its
> memory for disk caching and then tell postgres about it.
>

Because you can't.  It already uses ALL RAM that isn't in use for
something else.


Re: FreeBSD config

From
Christopher Weimann
Date:
On 02/26/2004-11:16AM, Dror Matalon wrote:
> >
> > effective_cache_size changes no cache settings for postgresql, it simply
> > acts as a hint to the planner on about how much of the dataset your OS /
> > Kernel / Disk cache can hold.
>
> I understand that. The question is why have the OS, in this case FreeBsd
> use only 200 Megs for disk cache and not more. Why not double the
> vfs.hibufspace  to 418119680 and double the effective_cache_size to 51040.
>

FreeBSD uses ALL ram that isn't being used for something else as
its disk cache.  The "effective_cache_size" in the PostGreSQL config
has no effect on how the OS chooses to use memory, it is just hint
to the PostGreSQL planner so it can guess the the likelyhood of
what it is looking for being in the cache.


Re: FreeBSD config

From
Mark Kirkwood
Date:

Shridhar Daithankar wrote:

> Dror Matalon wrote:
>
>> I've read Matt Dillon's discussion about the freebsd VM at
>> http://www.daemonnews.org/200001/freebsd_vm.html and I didn't see him
>> saying that Freebsd uses all the free RAM for disk cache. Would you care
>> to provide a URL pointing to that?
>
>
>
Quoting from http://www.daemonnews.org/200001/freebsd_vm.html :

<snip>*
When To Free a Page*

Since the VM system uses all available memory for disk caching, there
are usually very few truly-free pages...
</snip>

Got to say - a very interesting discussion you have all being having, I
am now quite confused about what those vfs.*buf* variables actually do...

Please feed back any clarfications from the FreeBSD experts to this list!

regards

Mark


Re: FreeBSD config

From
Dror Matalon
Date:
On Mon, Mar 01, 2004 at 08:30:58PM +1300, Mark Kirkwood wrote:
>
>
> Shridhar Daithankar wrote:
>
> >Dror Matalon wrote:
> >
> >>I've read Matt Dillon's discussion about the freebsd VM at
> >>http://www.daemonnews.org/200001/freebsd_vm.html and I didn't see him
> >>saying that Freebsd uses all the free RAM for disk cache. Would you care
> >>to provide a URL pointing to that?
> >
> >
> >

I noticed this passage too, but ...
> Quoting from http://www.daemonnews.org/200001/freebsd_vm.html :
>
> <snip>*
> When To Free a Page*
>
> Since the VM system uses all available memory for disk caching, there
        ^^^^^^^^^^^^^

The VM system, as you can see from the article, is focused on paging and
caching the programs and program data. Is the cache for disk reads and
writes thrown into the mix as well?

> are usually very few truly-free pages...
> </snip>

>
> Got to say - a very interesting discussion you have all being having, I
> am now quite confused about what those vfs.*buf* variables actually do...

Same here.

>
> Please feed back any clarfications from the FreeBSD experts to this list!
>
> regards
>
> Mark
>

--
Dror Matalon
Zapatec Inc
1700 MLK Way
Berkeley, CA 94709
http://www.fastbuzz.com
http://www.zapatec.com

Re: FreeBSD config

From
Mark Kirkwood
Date:

>I noticed this passage too, but ...
>
>
>>Quoting from http://www.daemonnews.org/200001/freebsd_vm.html :
>>
>><snip>*
>>When To Free a Page*
>>
>>Since the VM system uses all available memory for disk caching, there
>>
>>
>        ^^^^^^^^^^^^^
>
>The VM system, as you can see from the article, is focused on paging and
>caching the programs and program data. Is the cache for disk reads and
>writes thrown into the mix as well?
>
>
>
Yes - that is the real question. The following link :

http://www.freebsd.org/doc/en_US.ISO8859-1/books/arch-handbook/vm-cache.html

and the few "next" pages afterward talk about a unified buffer cache i.e
file buffer cache is part of the KVM system which is part of the VM
system - but this does not seem to preclude those vfs.*buf* variables
limiting the size of the file buffer cache... hmm ... so no real
decrease of confusion at this end.. :-)

Mark



Re: FreeBSD config

From
Vivek Khera
Date:
>>>>> "CW" == Christopher Weimann <cweimann@k12hq.com> writes:

>> which brings me back to my question why not make Freebsd use more of its
>> memory for disk caching and then tell postgres about it.
>>

CW> Because you can't.  It already uses ALL RAM that isn't in use for
CW> something else.

No, it does not.

--
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Vivek Khera, Ph.D.                Khera Communications, Inc.
Internet: khera@kciLink.com       Rockville, MD  +1-301-869-4449 x806
AIM: vivekkhera Y!: vivek_khera   http://www.khera.org/~vivek/