Thread: Huge pages not working on PG 9.4

Huge pages not working on PG 9.4

From
Michael Heaney
Date:
I've compiled and installed PG 9.4.0 on my Linux VM, but attempts to
enable huge pages (huge_pages=on in postgresql.conf) have resulted in
this error message in the log:

   FATAL:  huge pages not supported on this platform

I'm running CentOS 6.4.  Here's the uname output:

   Linux 2.6.32-358.2.1.el6.x86_64 #1 SMP Wed Mar 13 00:26:49 UTC 2013
x86_64 x86_64 x86_64 GNU/Linux

I've set a reasonable nr_hugepages value in /etc/sysctl.conf.  Is there
some kernel mod or environment variable that I also need?


Michael Heaney
JCVI



Re: Huge pages not working on PG 9.4

From
Scott Ribe
Date:
On Jan 16, 2015, at 3:07 PM, Michael Heaney <mheaney@jcvi.org> wrote:
>
> I've set a reasonable nr_hugepages value in /etc/sysctl.conf.  Is there some kernel mod or environment variable that
Ialso need? 

This?


<https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/5/html/Tuning_and_Optimizing_Red_Hat_Enterprise_Linux_for_Oracle_9i_and_10g_Databases/sect-Oracle_9i_and_10g_Tuning_Guide-Large_Memory_Optimization_Big_Pages_and_Huge_Pages-Configuring_Huge_Pages_in_Red_Hat_Enterprise_Linux_4_or_5.html>

--
Scott Ribe
scott_ribe@elevated-dev.com
http://www.elevated-dev.com/
(303) 722-0567 voice






Re: Huge pages not working on PG 9.4

From
Michael Heaney
Date:
On 1/16/2015 5:13 PM, Scott Ribe wrote:
> On Jan 16, 2015, at 3:07 PM, Michael Heaney <mheaney@jcvi.org> wrote:
>> I've set a reasonable nr_hugepages value in /etc/sysctl.conf.  Is there some kernel mod or environment variable that
Ialso need? 
> This?
>
>
<https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/5/html/Tuning_and_Optimizing_Red_Hat_Enterprise_Linux_for_Oracle_9i_and_10g_Databases/sect-Oracle_9i_and_10g_Tuning_Guide-Large_Memory_Optimization_Big_Pages_and_Huge_Pages-Configuring_Huge_Pages_in_Red_Hat_Enterprise_Linux_4_or_5.html>
>


I'll have the SA's add the 'memlock' parameter to /etc/security/limits.conf,
and try it again.  Thanks!

--
Michael Heaney
JCVI



Re: Huge pages not working on PG 9.4

From
Michael Heaney
Date:
On 1/16/2015 5:34 PM, Michael Heaney wrote:
> ...
>
> I'll have the SA's add the 'memlock' parameter to
> /etc/security/limits.conf,
> and try it again.  Thanks!
>
> -

<sigh>  Nope, that didn't make any difference.  Since I compile my own
binary,
I'm wondering if there's something that 'configure' didn't set properly.

--
Michael Heaney
JCVI



Re: Huge pages not working on PG 9.4

From
Scott Ribe
Date:
On Jan 16, 2015, at 3:50 PM, Michael Heaney <mheaney@jcvi.org> wrote:
>
> Nope, that didn't make any difference.  Since I compile my own binary,
> I'm wondering if there's something that 'configure' didn't set properly.

Possibly configure needed to see the settings you just made? (I really have idea--that first link I posted exhausted my
knowledgeon that subject.) 

--
Scott Ribe
scott_ribe@elevated-dev.com
http://www.elevated-dev.com/
(303) 722-0567 voice






Re: Huge pages not working on PG 9.4

From
Tom Lane
Date:
Michael Heaney <mheaney@jcvi.org> writes:
>> FATAL:  huge pages not supported on this platform

> <sigh>  Nope, that didn't make any difference.  Since I compile my own
> binary,
> I'm wondering if there's something that 'configure' didn't set properly.

A quick look at the PG source code says that that message means that
MAP_HUGETLB isn't defined.  Our code is expecting <sys/mman.h> to provide
that symbol, which it should do by default according to what I can see on
my RHEL 6.6 box.  Perhaps you've defined some of the feature restriction
macros like _POSIX_SOURCE?

            regards, tom lane


Re: Huge pages not working on PG 9.4

From
Michael Heaney
Date:
On 1/16/2015 6:58 PM, Tom Lane wrote:
> Michael Heaney <mheaney@jcvi.org> writes:
>>> FATAL:  huge pages not supported on this platform
>> <sigh>  Nope, that didn't make any difference.  Since I compile my own
>> binary,
>> I'm wondering if there's something that 'configure' didn't set properly.
> A quick look at the PG source code says that that message means that
> MAP_HUGETLB isn't defined.  Our code is expecting <sys/mman.h> to provide
> that symbol, which it should do by default according to what I can see on
> my RHEL 6.6 box.  Perhaps you've defined some of the feature restriction
> macros like _POSIX_SOURCE?
>
>             regards, tom lane

Thanks for the helpful clue, Tom.  I haven't knowingly defined
_POSIX_SOURCE or any other macros. I go through the same configure/make
process on the Centos 6.6 VM on my home PC, and huge pages works just
fine.  Maybe it's an issue with CentOS  6.4?

--
Michael Heaney
JCVI




Re: Huge pages not working on PG 9.4

From
Michael Heaney
Date:
On 1/16/2015 6:39 PM, Scott Ribe wrote:
> On Jan 16, 2015, at 3:50 PM, Michael Heaney <mheaney@jcvi.org> wrote:
>> Nope, that didn't make any difference.  Since I compile my own binary,
>> I'm wondering if there's something that 'configure' didn't set properly.
> Possibly configure needed to see the settings you just made? (I really have idea--that first link I posted exhausted
myknowledge on that subject.) 
>

I re-ran configure/make after rebooting the box, with no difference.  Thanks
for responding, though.  I recall reading that same link a couple years
ago,
when I was trying to get huge pages to work for Sybase ASE.

--
Michael Heaney
JCVI



Re: Huge pages not working on PG 9.4

From
Tom Lane
Date:
Michael Heaney <mheaney@jcvi.org> writes:
> On 1/16/2015 6:58 PM, Tom Lane wrote:
>> A quick look at the PG source code says that that message means that
>> MAP_HUGETLB isn't defined.  Our code is expecting <sys/mman.h> to provide
>> that symbol, which it should do by default according to what I can see on
>> my RHEL 6.6 box.  Perhaps you've defined some of the feature restriction
>> macros like _POSIX_SOURCE?

> Thanks for the helpful clue, Tom.  I haven't knowingly defined
> _POSIX_SOURCE or any other macros. I go through the same configure/make
> process on the Centos 6.6 VM on my home PC, and huge pages works just
> fine.  Maybe it's an issue with CentOS  6.4?

Dunno, hard to believe that hugetlb support wasn't there in 6.4.  A quick
troll through the RHEL 6.5 and 6.6 release notes finds something about
better hugetlb support for System Z processors, but what we're discussing
here shouldn't be architecture-specific.

You might try comparing /usr/include/sys/mman.h, /usr/include/bits/mman.h,
and /usr/include/features.h between your 6.4 and 6.6 systems to see if
you can spot any relevant differences.

            regards, tom lane


Re: Huge pages not working on PG 9.4

From
Michael Heaney
Date:
On 1/16/2015 10:52 PM, Tom Lane wrote:
> Michael Heaney <mheaney@jcvi.org> writes:
>> On 1/16/2015 6:58 PM, Tom Lane wrote:
>>> A quick look at the PG source code says that that message means that
>>> MAP_HUGETLB isn't defined.  Our code is expecting <sys/mman.h> to provide
>>> that symbol, which it should do by default according to what I can see on
>>> my RHEL 6.6 box.  Perhaps you've defined some of the feature restriction
>>> macros like _POSIX_SOURCE?
>> Thanks for the helpful clue, Tom.  I haven't knowingly defined
>> _POSIX_SOURCE or any other macros. I go through the same configure/make
>> process on the Centos 6.6 VM on my home PC, and huge pages works just
>> fine.  Maybe it's an issue with CentOS  6.4?
> Dunno, hard to believe that hugetlb support wasn't there in 6.4.  A quick
> troll through the RHEL 6.5 and 6.6 release notes finds something about
> better hugetlb support for System Z processors, but what we're discussing
> here shouldn't be architecture-specific.
>
> You might try comparing /usr/include/sys/mman.h, /usr/include/bits/mman.h,
> and /usr/include/features.h between your 6.4 and 6.6 systems to see if
> you can spot any relevant differences.
>
>             regards, tom lane

Nice catch!  The following line was missing from /usr/include/bits/mman.h
on CentOS 6.4:

# define MAP_HUGETLB    0x40000         /* Create huge page mapping.  */

I added it to a test 6.4 box, and the rebuilt Postgres was then able to
use huge pages.

I'm leery about adding this to all the 6.4 boxes, though, without some
reassurance that it won't break something. I'll bring it up with my
Linux SA's next week.  Thanks again for  your help!

--
Michael Heaney
JCVI





Re: Huge pages not working on PG 9.4

From
Glyn Astill
Date:
> From: Scott Ribe <scott_ribe@elevated-dev.com>


> To: Michael Heaney <mheaney@jcvi.org>
> Cc: "pgsql-admin@postgresql.org" <pgsql-admin@postgresql.org>
> Sent: Friday, 16 January 2015, 23:39
> Subject: Re: [ADMIN] Huge pages not working on PG 9.4
>
> On Jan 16, 2015, at 3:50 PM, Michael Heaney <mheaney@jcvi.org> wrote:
>>
>>  Nope, that didn't make any difference.  Since I compile my own binary,
>>  I'm wondering if there's something that 'configure'
> didn't set properly.
>
> Possibly configure needed to see the settings you just made? (I really have
> idea--that first link I posted exhausted my knowledge on that subject.)
>


I've very thin knowledge on this too, but I'll chip in with what I remember.  Not sure if you'd have to set
hugetlb_shm_groupto the gid of the postgres user. 

And I think you can see what support your kernel was compiled with this:

$ grep HUGE /boot/config-$(uname -r)

And a quick peek at what you've got set up currently:

$ grep -R "" /sys/kernel/mm/hugepages/ /proc/sys/vm/*huge*

$ cat /proc/meminfo |grep Huge


Perhaps if you post those to the list it'd help someone advise?