Thread: [doc fix] Correct calculation of vm.nr_hugepages
Hello, The current PostgreSQL documentation overestimates the number of huge pages (vm.nr_hugepages) because the calculation usesthe maximum virtual address space. In practice, huge pages are only used for the anonymous shared memory segment. Theattached patch fixes the documentation. FYI, Oracle presents a shell script to calculate the number of huge pages for its shared memory segments: https://docs.oracle.com/cd/E11882_01/server.112/e10839/appi_vlm.htm#UNXAR385 Regards Takayuki Tsunakawa
Attachment
On Mon, Feb 19, 2018 at 07:05:47AM +0000, Tsunakawa, Takayuki wrote:
> The current PostgreSQL documentation overestimates the number of huge pages
> (vm.nr_hugepages) because the calculation uses the maximum virtual address
> space. In practice, huge pages are only used for the anonymous shared memory
> segment. The attached patch fixes the documentation.
+ <userinput>pmap 4170 | grep rw-s | grep zero | awk '{print $2}'</userinput>
One can do with fewer greps:
pryzbyj@pryzbyj:~$ sudo pmap `pgrep -P 1 -u postgres` |awk '/rw-s/&&/zero/{print $2}' # check PPID=1
144848K
or:
pryzbyj@pryzbyj:~$ sudo pmap `pgrep -u postgres |sed q` |awk '/rw-s/&&/zero/{print $2}' # check any backend, not just
postmasterparent
144848K
Or (this is even less clean but gives an alternative which continues to use
/proc directly):
pryzbyj@pryzbyj:~$ sudo cat /proc/`pgrep -P 1 -u postgres`/maps |awk --non-decimal-data 'BEGIN{FS="[ -]"}
/zero/{a="0x"$1;b="0x"$2;print(b-a)/1024"kB"}'
144848kB
BTW when huge_pages=try, I've been verifying hugepages are in use by running:
pryzbyj@pryzbyj:~$ sudo sh -c 'cat /proc/`pgrep -u postgres -P1`/smaps |grep -c "KernelPageSize: *2048 kB"' || echo NOT
FOUND
0
NOT FOUND
Justin
From: Justin Pryzby [mailto:pryzby@telsasoft.com]
> One can do with fewer greps:
> pryzbyj@pryzbyj:~$ sudo pmap `pgrep -P 1 -u postgres` |awk
> '/rw-s/&&/zero/{print $2}' # check PPID=1 144848K
Thanks, I'd like to take this.
Regards
Takayuki Tsunakawa
Attachment
On Mon, Feb 19, 2018 at 9:43 PM, Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com> wrote: > Thanks, I'd like to take this. Why are these values so large? The example in the documentation shows 6490428 kB, and in my test I got 8733888 kB. But 8733888 kB = 8.3 TB! 8.3 GB would make sense, but 8.3 TB does not. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
On Wed, Feb 21, 2018 at 03:14:57PM -0500, Robert Haas wrote:
> On Mon, Feb 19, 2018 at 9:43 PM, Tsunakawa, Takayuki
> <tsunakawa.takay@jp.fujitsu.com> wrote:
> > Thanks, I'd like to take this.
>
> Why are these values so large? The example in the documentation shows
> 6490428 kB, and in my test I got 8733888 kB. But 8733888 kB = 8.3 TB!
> 8.3 GB would make sense, but 8.3 TB does not.
pryzbyj@pryzbyj:~$ units -t -v 8733888kB GiB
8733888kB = 8.1340671 GiB
On Wed, Feb 21, 2018 at 3:28 PM, Justin Pryzby <pryzby@telsasoft.com> wrote: > On Wed, Feb 21, 2018 at 03:14:57PM -0500, Robert Haas wrote: >> On Mon, Feb 19, 2018 at 9:43 PM, Tsunakawa, Takayuki >> <tsunakawa.takay@jp.fujitsu.com> wrote: >> > Thanks, I'd like to take this. >> >> Why are these values so large? The example in the documentation shows >> 6490428 kB, and in my test I got 8733888 kB. But 8733888 kB = 8.3 TB! >> 8.3 GB would make sense, but 8.3 TB does not. > > pryzbyj@pryzbyj:~$ units -t -v 8733888kB GiB > 8733888kB = 8.1340671 GiB Sigh. It would be nice if I were less stupid. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
The following review has been posted through the commitfest application: make installcheck-world: not tested Implements feature: not tested Spec compliant: not tested Documentation: tested, passed It works. Can be Merged.
On 2018-02-20 02:43:32 +0000, Tsunakawa, Takayuki wrote:
> diff --git a/doc/src/sgml/runtime.sgml b/doc/src/sgml/runtime.sgml
> index d162acb..1aed070 100644
> --- a/doc/src/sgml/runtime.sgml
> +++ b/doc/src/sgml/runtime.sgml
> @@ -1472,14 +1472,14 @@ export PG_OOM_ADJUST_VALUE=0
> the kernel setting <varname>vm.nr_hugepages</varname>. To estimate the
> number of huge pages needed, start <productname>PostgreSQL</productname>
> without huge pages enabled and check the
> - postmaster's <varname>VmPeak</varname> value, as well as the system's
> + postmaster's anonymous shared memory segment size, as well as the system's
> huge page size, using the <filename>/proc</filename> file system. This might
> look like:
> <programlisting>
> $ <userinput>head -1 $PGDATA/postmaster.pid</userinput>
> 4170
> -$ <userinput>grep ^VmPeak /proc/4170/status</userinput>
> -VmPeak: 6490428 kB
> +$ <userinput>pmap 4170 | awk '/rw-s/ && /zero/ {print $2}'</userinput>
> +6490428K
> $ <userinput>grep ^Hugepagesize /proc/meminfo</userinput>
> Hugepagesize: 2048 kB
> </programlisting>
One disadvantage of this is that it relies on the presence of pmap,
which IIRC is not installed by default in a number of distributions. Are
we concerned about that?
Greetings,
Andres Freund
On 2/26/18 08:25, Vasundhar Boddapati wrote: > The following review has been posted through the commitfest application: > make installcheck-world: not tested > Implements feature: not tested > Spec compliant: not tested > Documentation: tested, passed > > It works. Can be Merged. committed -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
On 3/1/18 04:54, Andres Freund wrote: > One disadvantage of this is that it relies on the presence of pmap, > which IIRC is not installed by default in a number of distributions. Are > we concerned about that? It's in the same package as "top", so I think that's fine. -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services