Thread: OS X shared memory documentation
I finally figured out what the issue was with OS X blowing away shared memory settings from /etc/sysctl.conf. I added a note to the 8.1 docs already, and here's a patch against -HEAD. -- Jim C. Nasby, Sr. Engineering Consultant jnasby@pervasive.com Pervasive Software http://pervasive.com work: 512-231-6117 vcard: http://jim.nasby.net/pervasive.vcf cell: 512-569-9461
Attachment
Thanks for this Jim! We'll test it and report results. - Luke ________________________________ From: pgsql-patches-owner@postgresql.org on behalf of Jim C. Nasby Sent: Sat 2/11/2006 9:26 PM To: pgsql-patches@postgresql.org Subject: [PATCHES] OS X shared memory documentation I finally figured out what the issue was with OS X blowing away shared memory settings from /etc/sysctl.conf. I added a note to the 8.1 docs already, and here's a patch against -HEAD. -- Jim C. Nasby, Sr. Engineering Consultant jnasby@pervasive.com Pervasive Software http://pervasive.com work: 512-231-6117 vcard: http://jim.nasby.net/pervasive.vcf cell: 512-569-9461
"Jim C. Nasby" <jnasby@pervasive.com> writes:
> + In OS X 10.3 and later, you can simply place the settings into
> + <filename>/etc/sysctl.conf</>, with one caveat: all five of these
> + parameters are linked together, so you must specify all five in
> + <filename>/etc/sysctl.conf</>. If you do not specify all five the
> + defaults that are hard-coded into <filename>/etc/rc</> will be used
> + instead. In all versions, you'll need to reboot to make
It'd be nice if it were as simple as that, but my experiments say that
it isn't.
The conclusion I had come to (which is in our docs now) is that the real
gotcha in the mix is that 10.4 ignores SHMMAX settings that aren't an
exact multiple of the page size.
regards, tom lane
I wrote: > It'd be nice if it were as simple as that, but my experiments say that > it isn't. It took me a little bit to dig up the results of my previous experiments, but here they are: http://archives.postgresql.org/pgsql-hackers/2005-10/msg00684.php AFAICT, setting all vs some of the variables isn't a factor; at least not in 10.4 (couldn't say about 10.3 anymore). regards, tom lane
The definitive answer can be found in the source code for the Darwin
kernel, xnu (links are below):
1. The shmmax, shmmin, shmmni, shmseg, and shmall settings cannot be
changed after the shared memory system is initialized
2. The shared memory system is initialized immediately after all 5
settings have been configured
3. The shmall setting must be a multiple of the page size (on both
10.3 and 10.4)
Here are excerpts from the sysctl_shminfo() function found in
sysv_shm.c:
[1]
/* Set the values only if shared memory is not initialised */
if (!shm_inited) {
if ((error = SYSCTL_IN(req, arg1, sizeof(user_ssize_t)))
!= 0) {
sysctl_shminfo_ret = error;
goto sysctl_shminfo_out;
}
[2]
/* Initialize only when all values are set */
if ((shminfo.shmmax != (user_ssize_t)-1) &&
(shminfo.shmmin != (user_ssize_t)-1) &&
(shminfo.shmmni != (user_ssize_t)-1) &&
(shminfo.shmseg != (user_ssize_t)-1) &&
(shminfo.shmall != (user_ssize_t)-1)) {
shminit(NULL);
}
[3]
(10.3)
if (arg1 == &shminfo.shmmax) {
if (shminfo.shmmax & PAGE_MASK) {
shminfo.shmmax = -1;
return(EINVAL);
}
}
(10.4)
if (arg1 == &shminfo.shmmax) {
if (shminfo.shmmax & PAGE_MASK_64) {
shminfo.shmmax = -1;
return(EINVAL);
}
}
PAGE_MASK is "(PAGE_SIZE - 1)", and PAGE_MASK_64 is simply "(unsigned
long long)PAGE_MASK"
/etc/rc contains commands to set all 5 of the parameters. In 10.3.9
and later, you can place your own customized settings inside /etc/
sysctl.conf that will be executed before /etc/rc's commands. But if
you don't set them all, the shared memory system isn't initialized,
and /etc/rc will overwrite them.
Does that make sense?
In a nutshell, if you want to customize the shared memory settings,
you must:
(On 10.3.9 and later)
Set ALL 5 settings inside /etc/sysctl.conf, and make sure that shmmax
is a multiple of the page size (4096?). The page size is a runtime
variable that's set by kernel bootstrap mechanisms calling
vm_set_page_size() (I haven't tracked those down yet).
(On earlier systems)
Modify the desired settings inside /etc/rc, and be aware that OS
updates will overwrite your changes. If you upgrade to 10.3.9,
migrate to using /etc/sysctl.conf.
In my own OS X products that use a PostgreSQL server, I install an /
etc/sysctl.conf file on all servers. If installing on anything
earlier than 10.3, my installer modifies /etc/rc. That way, when the
customer upgrades their server to 10.3.9 and the /etc/rc file is
overwritten, the /etc/sysctl.conf file is already in place.
Thanks!
- Chris
Referenced source code:
Mac OS X 10.4.4:
http://www.opensource.apple.com/darwinsource/10.4.4.ppc/
xnu-792.6.56/bsd/kern/sysv_shm.c
Mac OS X 10.3.9:
http://www.opensource.apple.com/darwinsource/10.3.9/xnu-517.12.7/
bsd/kern/sysv_shm.c
Mac OS X 10.3:
http://www.opensource.apple.com/darwinsource/10.3/xnu-517/bsd/
kern/sysv_shm.c
Attachment
Outstanding writeup! Mind if I post it in our knowledgebase at http://www.pervasivepostgres.com/instantkb13/? (Unless someone has a better idea of where this could go...) On Sun, Feb 12, 2006 at 04:33:27PM -0500, Chris Campbell wrote: > The definitive answer can be found in the source code for the Darwin > kernel, xnu (links are below): > > 1. The shmmax, shmmin, shmmni, shmseg, and shmall settings cannot be > changed after the shared memory system is initialized > > 2. The shared memory system is initialized immediately after all 5 > settings have been configured > > 3. The shmall setting must be a multiple of the page size (on both > 10.3 and 10.4) > > Here are excerpts from the sysctl_shminfo() function found in > sysv_shm.c: > > [1] > /* Set the values only if shared memory is not initialised */ > if (!shm_inited) { > if ((error = SYSCTL_IN(req, arg1, sizeof(user_ssize_t))) > != 0) { > sysctl_shminfo_ret = error; > goto sysctl_shminfo_out; > } > > [2] > /* Initialize only when all values are set */ > if ((shminfo.shmmax != (user_ssize_t)-1) && > (shminfo.shmmin != (user_ssize_t)-1) && > (shminfo.shmmni != (user_ssize_t)-1) && > (shminfo.shmseg != (user_ssize_t)-1) && > (shminfo.shmall != (user_ssize_t)-1)) { > shminit(NULL); > } > > > [3] > (10.3) > if (arg1 == &shminfo.shmmax) { > if (shminfo.shmmax & PAGE_MASK) { > shminfo.shmmax = -1; > return(EINVAL); > } > } > > (10.4) > if (arg1 == &shminfo.shmmax) { > if (shminfo.shmmax & PAGE_MASK_64) { > shminfo.shmmax = -1; > return(EINVAL); > } > } > > PAGE_MASK is "(PAGE_SIZE - 1)", and PAGE_MASK_64 is simply "(unsigned > long long)PAGE_MASK" > > > /etc/rc contains commands to set all 5 of the parameters. In 10.3.9 > and later, you can place your own customized settings inside /etc/ > sysctl.conf that will be executed before /etc/rc's commands. But if > you don't set them all, the shared memory system isn't initialized, > and /etc/rc will overwrite them. > > Does that make sense? > > In a nutshell, if you want to customize the shared memory settings, > you must: > > (On 10.3.9 and later) > Set ALL 5 settings inside /etc/sysctl.conf, and make sure that shmmax > is a multiple of the page size (4096?). The page size is a runtime > variable that's set by kernel bootstrap mechanisms calling > vm_set_page_size() (I haven't tracked those down yet). > > (On earlier systems) > Modify the desired settings inside /etc/rc, and be aware that OS > updates will overwrite your changes. If you upgrade to 10.3.9, > migrate to using /etc/sysctl.conf. > > In my own OS X products that use a PostgreSQL server, I install an / > etc/sysctl.conf file on all servers. If installing on anything > earlier than 10.3, my installer modifies /etc/rc. That way, when the > customer upgrades their server to 10.3.9 and the /etc/rc file is > overwritten, the /etc/sysctl.conf file is already in place. > > Thanks! > > - Chris > > > Referenced source code: > > Mac OS X 10.4.4: > http://www.opensource.apple.com/darwinsource/10.4.4.ppc/ > xnu-792.6.56/bsd/kern/sysv_shm.c > > Mac OS X 10.3.9: > http://www.opensource.apple.com/darwinsource/10.3.9/xnu-517.12.7/ > bsd/kern/sysv_shm.c > > Mac OS X 10.3: > http://www.opensource.apple.com/darwinsource/10.3/xnu-517/bsd/ > kern/sysv_shm.c > -- Jim C. Nasby, Sr. Engineering Consultant jnasby@pervasive.com Pervasive Software http://pervasive.com work: 512-231-6117 vcard: http://jim.nasby.net/pervasive.vcf cell: 512-569-9461
Jim C. Nasby wrote: > Outstanding writeup! Mind if I post it in our knowledgebase at > http://www.pervasivepostgres.com/instantkb13/? (Unless someone has a > better idea of where this could go...) Some of this information should be added to the documentation on how to configure the shared memory limits for the different Unixen (16.4.1 of the main docs). -Neil
Great. I have applied the following documentation patch for CVS HEAD
and 8.1.X.
It seems you are saying that shmmax and shmall must be multiples of 4k,
and I added that to the docs. It previously only mentioned shmmax in
that regard.
---------------------------------------------------------------------------
Chris Campbell wrote:
> The definitive answer can be found in the source code for the Darwin
> kernel, xnu (links are below):
>
> 1. The shmmax, shmmin, shmmni, shmseg, and shmall settings cannot be
> changed after the shared memory system is initialized
>
> 2. The shared memory system is initialized immediately after all 5
> settings have been configured
>
> 3. The shmall setting must be a multiple of the page size (on both
> 10.3 and 10.4)
>
> Here are excerpts from the sysctl_shminfo() function found in
> sysv_shm.c:
>
> [1]
> /* Set the values only if shared memory is not initialised */
> if (!shm_inited) {
> if ((error = SYSCTL_IN(req, arg1, sizeof(user_ssize_t)))
> != 0) {
> sysctl_shminfo_ret = error;
> goto sysctl_shminfo_out;
> }
>
> [2]
> /* Initialize only when all values are set */
> if ((shminfo.shmmax != (user_ssize_t)-1) &&
> (shminfo.shmmin != (user_ssize_t)-1) &&
> (shminfo.shmmni != (user_ssize_t)-1) &&
> (shminfo.shmseg != (user_ssize_t)-1) &&
> (shminfo.shmall != (user_ssize_t)-1)) {
> shminit(NULL);
> }
>
>
> [3]
> (10.3)
> if (arg1 == &shminfo.shmmax) {
> if (shminfo.shmmax & PAGE_MASK) {
> shminfo.shmmax = -1;
> return(EINVAL);
> }
> }
>
> (10.4)
> if (arg1 == &shminfo.shmmax) {
> if (shminfo.shmmax & PAGE_MASK_64) {
> shminfo.shmmax = -1;
> return(EINVAL);
> }
> }
>
> PAGE_MASK is "(PAGE_SIZE - 1)", and PAGE_MASK_64 is simply "(unsigned
> long long)PAGE_MASK"
>
>
> /etc/rc contains commands to set all 5 of the parameters. In 10.3.9
> and later, you can place your own customized settings inside /etc/
> sysctl.conf that will be executed before /etc/rc's commands. But if
> you don't set them all, the shared memory system isn't initialized,
> and /etc/rc will overwrite them.
>
> Does that make sense?
>
> In a nutshell, if you want to customize the shared memory settings,
> you must:
>
> (On 10.3.9 and later)
> Set ALL 5 settings inside /etc/sysctl.conf, and make sure that shmmax
> is a multiple of the page size (4096?). The page size is a runtime
> variable that's set by kernel bootstrap mechanisms calling
> vm_set_page_size() (I haven't tracked those down yet).
>
> (On earlier systems)
> Modify the desired settings inside /etc/rc, and be aware that OS
> updates will overwrite your changes. If you upgrade to 10.3.9,
> migrate to using /etc/sysctl.conf.
>
> In my own OS X products that use a PostgreSQL server, I install an /
> etc/sysctl.conf file on all servers. If installing on anything
> earlier than 10.3, my installer modifies /etc/rc. That way, when the
> customer upgrades their server to 10.3.9 and the /etc/rc file is
> overwritten, the /etc/sysctl.conf file is already in place.
>
> Thanks!
>
> - Chris
>
>
> Referenced source code:
>
> Mac OS X 10.4.4:
> http://www.opensource.apple.com/darwinsource/10.4.4.ppc/
> xnu-792.6.56/bsd/kern/sysv_shm.c
>
> Mac OS X 10.3.9:
> http://www.opensource.apple.com/darwinsource/10.3.9/xnu-517.12.7/
> bsd/kern/sysv_shm.c
>
> Mac OS X 10.3:
> http://www.opensource.apple.com/darwinsource/10.3/xnu-517/bsd/
> kern/sysv_shm.c
>
--
Bruce Momjian http://candle.pha.pa.us
SRA OSS, Inc. http://www.sraoss.com
+ If your life is a hard drive, Christ can be your backup. +
Index: doc/src/sgml/runtime.sgml
===================================================================
RCS file: /cvsroot/pgsql/doc/src/sgml/runtime.sgml,v
retrieving revision 1.361
diff -c -c -r1.361 runtime.sgml
*** doc/src/sgml/runtime.sgml 5 Jan 2006 03:01:33 -0000 1.361
--- doc/src/sgml/runtime.sgml 2 Mar 2006 20:28:06 -0000
***************
*** 886,904 ****
changes take effect.
</para>
! <!--
! It might appear that we could recommend putting sysctl settings into
! /etc/sysctl.conf so that they aren't destroyed by OS version updates.
! This does not work as of OS X 10.4.2, however, because /etc/rc
! promptly overrides whatever SHM settings are read from sysctl.conf :-(
! Perhaps someday we can recommend this ...
! -->
<para>
! <varname>SHMALL</> is measured in 4KB pages on this platform.
! Also note that some releases of OS X will reject attempts to
! set <varname>SHMMAX</> to a value that isn't an exact multiple
! of 4096.
</para>
</listitem>
</varlistentry>
--- 886,904 ----
changes take effect.
</para>
! <para>
! In OS X 10.3.9 and later, the file <filename>/etc/sysctl.conf</>
! allows shared memory setting to be saved across operating system
! upgrades, and is the recommended method for setting these
! parameters. When using this file, all five shared memory values
! must be set or the changes will be ignored.
! </para>
<para>
! <varname>SHMALL</> is measured in 4KB pages on this platform,
! and recent releases of OS X reject attempts to set
! <varname>SHMALL</> and <varname>SHMMAX</> to a value
! that isn't an exact multiple of 4096.
</para>
</listitem>
</varlistentry>
On Mar 2, 2006, at 15:32, Bruce Momjian wrote: > It seems you are saying that shmmax and shmall must be multiples of > 4k, > and I added that to the docs. It previously only mentioned shmmax in > that regard. That was a typo in my comment before the code (the nutshell descriptions after the code are correct). Only shmmax needs to be multiples of the page size (at least, that's how I interpret the Darwin code). Thanks! - Chris
Patch applied. Thanks. Your documentation changes can be viewed in five minutes using links on the developer's page, http://www.postgresql.org/developer/testing. --------------------------------------------------------------------------- Chris Campbell wrote: > On Mar 2, 2006, at 15:32, Bruce Momjian wrote: > > > It seems you are saying that shmmax and shmall must be multiples of > > 4k, > > and I added that to the docs. It previously only mentioned shmmax in > > that regard. > > That was a typo in my comment before the code (the nutshell > descriptions after the code are correct). Only shmmax needs to be > multiples of the page size (at least, that's how I interpret the > Darwin code). > > Thanks! > > - Chris > > > ---------------------------(end of broadcast)--------------------------- > TIP 6: explain analyze is your friend > -- Bruce Momjian http://candle.pha.pa.us SRA OSS, Inc. http://www.sraoss.com + If your life is a hard drive, Christ can be your backup. +