Thread: doc patch - linux memory handling
As discussed on Hackers.
enjoy
andrew
Index: doc/src/sgml/runtime.sgml
===================================================================
RCS file: /projects/cvsroot/pgsql-server/doc/src/sgml/runtime.sgml,v
retrieving revision 1.200
diff -c -w -r1.200 runtime.sgml
*** doc/src/sgml/runtime.sgml 17 Aug 2003 22:19:10 -0000 1.200
--- doc/src/sgml/runtime.sgml 21 Aug 2003 16:57:43 -0000
***************
*** 3068,3081 ****
</para>
<para>
! Linux has poor default memory overcommit behavior. Rather than
! failing if it can not reserve enough memory, it returns success,
! but later fails when the memory can't be mapped and terminates
! the application with <literal>kill -9</>. To prevent unpredictable
! process termination, use:
<programlisting>
sysctl -w vm.overcommit_memory=3
</programlisting>
Note, you will need enough swap space to cover all your memory needs.
</para>
</listitem>
--- 3068,3125 ----
</para>
<para>
! Linux kernel version 2.4.* has poor default memory overcommit
! behavior, which can result in the postmaster being killed by the
! kernel due to memory demands by another process if the system
! runs out of memory.
! </para>
!
! <para>
! The symptom of this occuring is a kernel message looking like
! this (consult your system documentation and configuration on
! where to look for such a message):
! <programlisting>
! Out of Memory: Killed process 12345 (postmaster).
! </programlisting>
! </para>
!
! <para>
! To avoid this situation, run postgres on a machine where you
! can be sure that other processes will not run the machine out
! of memory. If your kernel supports strict and/or paranoid modes
! of overcommit handling, you can also relieve this problem by
! altering the system's default behaviour. This can be determined
! by examining the function <varname>vm_enough_memory</>
! in the file <filename>mm/mmap.c</>in the kernel source.
! If this file reveals that strict and/or paranoid modes are
! supported by your kernel, turn one of these modes on by using
! <programlisting>
! sysctl -w vm.overcommit_memory=2
! </programlisting>
! for strict mode or
<programlisting>
sysctl -w vm.overcommit_memory=3
</programlisting>
+ for paranoid mode, or placing an equivalent entry in
+ <filename>/etc/sysctl.conf</>.
+ </para>
+
+ <para>
+ Warning: using these settings in a kernel which does not support
+ these modes will almost certainly increase the danger of the
+ kernel killing the postmaster, rather than reducing it.
+ If in any doubt, consult a kernel expert or your kernel vendor.
+ </para>
+
+ <para>
+ These modes are expected to be supported in all 2.6 and later
+ kernels. Some vendor 2.4 kernels may also support these modes.
+ However, it is known that some vendor documents suggest that
+ they support them while examination of the kernel source reveals
+ that they do not.
+ </para>
+
+ <para>
Note, you will need enough swap space to cover all your memory needs.
</para>
</listitem>
Minor SGML gripes... On Fri, Aug 22, 2003 at 10:00:00AM -0400, Andrew Dunstan wrote: > ! To avoid this situation, run postgres on a machine where you "postgres" should be "<productname>PostgreSQL</productname>". > ! altering the system's default behaviour. This can be determined > ! by examining the function <varname>vm_enough_memory</> The <function> tag is more appropriate. > + <para> > + Warning: using these settings in a kernel which does not support > + these modes will almost certainly increase the danger of the > + kernel killing the postmaster, rather than reducing it. > + If in any doubt, consult a kernel expert or your kernel vendor. > + </para> You should use <warning> or <note> tags for this paragraph. -Neil
I didn't find a single instance of a <warning> tag in the doc sources. I did find a warning in an unadorned para, in func.sgml. Anyway, I made the changes (using <note>) but anoncvs is now refusing my login attempt, so I can't do the diff. I've spent way too much time on this already - really my only concern was that the docs would give people info that would make things worse rather than better, and nobody else was doing anything about it. cheers andrew Neil Conway wrote: >Minor SGML gripes... > >On Fri, Aug 22, 2003 at 10:00:00AM -0400, Andrew Dunstan wrote: > > >>! To avoid this situation, run postgres on a machine where you >> >> > >"postgres" should be "<productname>PostgreSQL</productname>". > > > >>! altering the system's default behaviour. This can be determined >>! by examining the function <varname>vm_enough_memory</> >> >> > >The <function> tag is more appropriate. > > > >>+ <para> >>+ Warning: using these settings in a kernel which does not support >>+ these modes will almost certainly increase the danger of the >>+ kernel killing the postmaster, rather than reducing it. >>+ If in any doubt, consult a kernel expert or your kernel vendor. >>+ </para> >> >> > >You should use <warning> or <note> tags for this paragraph. > >-Neil > > >---------------------------(end of broadcast)--------------------------- >TIP 8: explain analyze is your friend > > >
It appears we are back in the "you need a non-empty password" mode.
Anyway, here's the revised patch.
andrew
Andrew Dunstan wrote:
>
> I didn't find a single instance of a <warning> tag in the doc sources.
> I did find a warning in an unadorned para, in func.sgml.
>
> Anyway, I made the changes (using <note>) but anoncvs is now refusing
> my login attempt, so I can't do the diff.
>
> I've spent way too much time on this already - really my only concern
> was that the docs would give people info that would make things worse
> rather than better, and nobody else was doing anything about it.
>
> cheers
>
> andrew
>
Index: doc/src/sgml/runtime.sgml
===================================================================
RCS file: /projects/cvsroot/pgsql-server/doc/src/sgml/runtime.sgml,v
retrieving revision 1.200
diff -c -w -r1.200 runtime.sgml
*** doc/src/sgml/runtime.sgml 17 Aug 2003 22:19:10 -0000 1.200
--- doc/src/sgml/runtime.sgml 22 Aug 2003 16:14:47 -0000
***************
*** 3068,3081 ****
</para>
<para>
! Linux has poor default memory overcommit behavior. Rather than
! failing if it can not reserve enough memory, it returns success,
! but later fails when the memory can't be mapped and terminates
! the application with <literal>kill -9</>. To prevent unpredictable
! process termination, use:
<programlisting>
sysctl -w vm.overcommit_memory=3
</programlisting>
Note, you will need enough swap space to cover all your memory needs.
</para>
</listitem>
--- 3068,3128 ----
</para>
<para>
! Linux kernel version 2.4.* has poor default memory overcommit
! behavior, which can result in the postmaster being killed by the
! kernel due to memory demands by another process if the system
! runs out of memory.
! </para>
!
! <para>
! The symptom of this occuring is a kernel message looking like
! this (consult your system documentation and configuration on
! where to look for such a message):
! <programlisting>
! Out of Memory: Killed process 12345 (postmaster).
! </programlisting>
! </para>
!
! <para>
! To avoid this situation, run <productname>PostgreSQL</productname>
! on a machine where you
! can be sure that other processes will not run the machine out
! of memory. If your kernel supports strict and/or paranoid modes
! of overcommit handling, you can also relieve this problem by
! altering the system's default behaviour. This can be determined
! by examining the function <function>vm_enough_memory</>
! in the file <filename>mm/mmap.c</>in the kernel source.
! If this file reveals that strict and/or paranoid modes are
! supported by your kernel, turn one of these modes on by using
! <programlisting>
! sysctl -w vm.overcommit_memory=2
! </programlisting>
! for strict mode or
<programlisting>
sysctl -w vm.overcommit_memory=3
</programlisting>
+ for paranoid mode, or placing an equivalent entry in
+ <filename>/etc/sysctl.conf</>.
+ </para>
+
+ <note>
+ <para>
+ Warning: using these settings in a kernel which does not support
+ these modes will almost certainly increase the danger of the
+ kernel killing the postmaster, rather than reducing it.
+ If in any doubt, consult a kernel expert or your kernel vendor.
+ </para>
+ </note>
+
+ <para>
+ These modes are expected to be supported in all 2.6 and later
+ kernels. Some vendor 2.4 kernels may also support these modes.
+ However, it is known that some vendor documents suggest that
+ they support them while examination of the kernel source reveals
+ that they do not.
+ </para>
+
+ <para>
Note, you will need enough swap space to cover all your memory needs.
</para>
</listitem>
Andrew Dunstan writes: > + <para> > + Warning: using these settings in a kernel which does not support > + these modes will almost certainly increase the danger of the > + kernel killing the postmaster, rather than reducing it. > + If in any doubt, consult a kernel expert or your kernel vendor. > + </para> If find this a bit unsatisfactory. If the kernel does not support these modes, then "using" them should have no effect. At least, since the kernel doesn't know what they mean, the danger of a postmaster kill cannot increase. Oh, and have you ever tried to contact a kernel expert or kernel vendor? :-) -- Peter Eisentraut peter_e@gmx.net
It's highly unsatisfactory, but it's a fact nonetheless. The stock kernel simply checks for a non-zero value in this variable and says "OK, give them the memory" if it finds it. So if you are using such a kernel, then setting the non-zero value turns off all checking, rather than using the default heuristic check. That's why advising people to use a value of, say, 3, is so dangerous. Go and read the source if you don't believe me. Actually, I've had some success with both vendors and experts. Far more than I have with vendors of proprietary software. But YMMV. The point, though, is that this is not our problem, as I see it. andrew Peter Eisentraut wrote: >Andrew Dunstan writes: > > > >>+ <para> >>+ Warning: using these settings in a kernel which does not support >>+ these modes will almost certainly increase the danger of the >>+ kernel killing the postmaster, rather than reducing it. >>+ If in any doubt, consult a kernel expert or your kernel vendor. >>+ </para> >> >> > >If find this a bit unsatisfactory. If the kernel does not support these >modes, then "using" them should have no effect. At least, since the >kernel doesn't know what they mean, the danger of a postmaster kill cannot >increase. > >Oh, and have you ever tried to contact a kernel expert or kernel vendor? >:-) > > >
Patch applied. Thanks. --------------------------------------------------------------------------- Andrew Dunstan wrote: > > It appears we are back in the "you need a non-empty password" mode. > > Anyway, here's the revised patch. > > andrew > > > Andrew Dunstan wrote: > > > > > I didn't find a single instance of a <warning> tag in the doc sources. > > I did find a warning in an unadorned para, in func.sgml. > > > > Anyway, I made the changes (using <note>) but anoncvs is now refusing > > my login attempt, so I can't do the diff. > > > > I've spent way too much time on this already - really my only concern > > was that the docs would give people info that would make things worse > > rather than better, and nobody else was doing anything about it. > > > > cheers > > > > andrew > > > > Index: doc/src/sgml/runtime.sgml > =================================================================== > RCS file: /projects/cvsroot/pgsql-server/doc/src/sgml/runtime.sgml,v > retrieving revision 1.200 > diff -c -w -r1.200 runtime.sgml > *** doc/src/sgml/runtime.sgml 17 Aug 2003 22:19:10 -0000 1.200 > --- doc/src/sgml/runtime.sgml 22 Aug 2003 16:14:47 -0000 > *************** > *** 3068,3081 **** > </para> > > <para> > ! Linux has poor default memory overcommit behavior. Rather than > ! failing if it can not reserve enough memory, it returns success, > ! but later fails when the memory can't be mapped and terminates > ! the application with <literal>kill -9</>. To prevent unpredictable > ! process termination, use: > <programlisting> > sysctl -w vm.overcommit_memory=3 > </programlisting> > Note, you will need enough swap space to cover all your memory needs. > </para> > </listitem> > --- 3068,3128 ---- > </para> > > <para> > ! Linux kernel version 2.4.* has poor default memory overcommit > ! behavior, which can result in the postmaster being killed by the > ! kernel due to memory demands by another process if the system > ! runs out of memory. > ! </para> > ! > ! <para> > ! The symptom of this occuring is a kernel message looking like > ! this (consult your system documentation and configuration on > ! where to look for such a message): > ! <programlisting> > ! Out of Memory: Killed process 12345 (postmaster). > ! </programlisting> > ! </para> > ! > ! <para> > ! To avoid this situation, run <productname>PostgreSQL</productname> > ! on a machine where you > ! can be sure that other processes will not run the machine out > ! of memory. If your kernel supports strict and/or paranoid modes > ! of overcommit handling, you can also relieve this problem by > ! altering the system's default behaviour. This can be determined > ! by examining the function <function>vm_enough_memory</> > ! in the file <filename>mm/mmap.c</>in the kernel source. > ! If this file reveals that strict and/or paranoid modes are > ! supported by your kernel, turn one of these modes on by using > ! <programlisting> > ! sysctl -w vm.overcommit_memory=2 > ! </programlisting> > ! for strict mode or > <programlisting> > sysctl -w vm.overcommit_memory=3 > </programlisting> > + for paranoid mode, or placing an equivalent entry in > + <filename>/etc/sysctl.conf</>. > + </para> > + > + <note> > + <para> > + Warning: using these settings in a kernel which does not support > + these modes will almost certainly increase the danger of the > + kernel killing the postmaster, rather than reducing it. > + If in any doubt, consult a kernel expert or your kernel vendor. > + </para> > + </note> > + > + <para> > + These modes are expected to be supported in all 2.6 and later > + kernels. Some vendor 2.4 kernels may also support these modes. > + However, it is known that some vendor documents suggest that > + they support them while examination of the kernel source reveals > + that they do not. > + </para> > + > + <para> > Note, you will need enough swap space to cover all your memory needs. > </para> > </listitem> > > ---------------------------(end of broadcast)--------------------------- > TIP 7: don't forget to increase your free space map settings -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania 19073
seems to be little syntax problem:
On Thu, 4 Sep 2003, Bruce Momjian wrote:
>
> Patch applied. Thanks.
>
> ---------------------------------------------------------------------------
>
>
> Andrew Dunstan wrote:
...
> >
> > ! <para>
> > ! To avoid this situation, run <productname>PostgreSQL</productname>
> > ! on a machine where you
> > ! can be sure that other processes will not run the machine out
> > ! of memory. If your kernel supports strict and/or paranoid modes
> > ! of overcommit handling, you can also relieve this problem by
> > ! altering the system's default behaviour. This can be determined
> > ! by examining the function <function>vm_enough_memory</>
> > ! in the file <filename>mm/mmap.c</>in the kernel source.
^^
--
Because strait is the gate, and narrow is the way, which leadeth unto
life, and few there be that find it. (MAT 7:14)
Ask, and it shall be given you; seek, and ye shall find;
knock, and it shall be opened unto you... (MAT 7:7)
ANT17-RIPE
http://www.imt.com.ua
Bruce, can you the missing space on the marked line, or do you need another patch from me? thanks andrew Andriy Tkachuk wrote: >seems to be little syntax problem: > >On Thu, 4 Sep 2003, Bruce Momjian wrote: > > > >>Patch applied. Thanks. >> >>--------------------------------------------------------------------------- >> >> >>Andrew Dunstan wrote: >> >> >... > > >>>! <para> >>>! To avoid this situation, run <productname>PostgreSQL</productname> >>>! on a machine where you >>>! can be sure that other processes will not run the machine out >>>! of memory. If your kernel supports strict and/or paranoid modes >>>! of overcommit handling, you can also relieve this problem by >>>! altering the system's default behaviour. This can be determined >>>! by examining the function <function>vm_enough_memory</> >>> >>> > > > > >>>! in the file <filename>mm/mmap.c</>in the kernel source. >>> >>> > ^^ >-- > Because strait is the gate, and narrow is the way, which leadeth unto > life, and few there be that find it. (MAT 7:14) > Ask, and it shall be given you; seek, and ye shall find; > knock, and it shall be opened unto you... (MAT 7:7) > >ANT17-RIPE > >http://www.imt.com.ua > > > >---------------------------(end of broadcast)--------------------------- >TIP 4: Don't 'kill -9' the postmaster > > >
Andrew Dunstan <andrew@dunslane.net> writes:
> can you the missing space on the marked line, or do you need another
> patch from me?
Fixed.
regards, tom lane