Thread: LOG: munmap(0x7fff80000000) failed: Invalid argument

Chris Richards <chris@infinite.io> writes:
> Setting up postgresql-9.5 (9.5.4-1.pgdg14.04+2) ...
> Creating new cluster 9.5/main ...
>   config /etc/postgresql/9.5/main
>   data   /var/lib/postgresql/9.5/main
>   locale en_US.UTF-8
> LOG:  munmap(0x7fff80000000) failed: Invalid argument
> [... snip 14 or so repeats ...]
> LOG:  munmap(0x7fff80000000) failed: Invalid argument
>   socket /var/run/postgresql
>   port   5433
> update-alternatives: using
> /usr/share/postgresql/9.5/man/man1/postmaster.1.gz to provide
> /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode
>  * Starting PostgreSQL 9.5 database server
>       [ OK ]
> Processing triggers for libc-bin (2.19-0ubuntu6.6) ...

> I'm able to connect and I dumped a few default relations.

> Is the munmap error of concern? It remains upon rebooting / restarting the
> server.

Seems pretty fishy to me; I don't know what would be causing it.

[ digs in code... ]  One theory is that PGSharedMemoryDetach is getting
called more than once, but I'm not sure how that would happen.  Can you
characterize where this happens more precisely?  What nondefault settings
have you got in postgresql.conf?

            regards, tom lane

On 2016-10-10 18:21:48 -0400, Tom Lane wrote:
> Chris Richards <chris@infinite.io> writes:
> > Setting up postgresql-9.5 (9.5.4-1.pgdg14.04+2) ...
> > Creating new cluster 9.5/main ...
> >   config /etc/postgresql/9.5/main
> >   data   /var/lib/postgresql/9.5/main
> >   locale en_US.UTF-8
> > LOG:  munmap(0x7fff80000000) failed: Invalid argument
> > [... snip 14 or so repeats ...]
> > LOG:  munmap(0x7fff80000000) failed: Invalid argument
> >   socket /var/run/postgresql
> >   port   5433
> > update-alternatives: using
> > /usr/share/postgresql/9.5/man/man1/postmaster.1.gz to provide
> > /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode
> >  * Starting PostgreSQL 9.5 database server
> >       [ OK ]
> > Processing triggers for libc-bin (2.19-0ubuntu6.6) ...
>
> > I'm able to connect and I dumped a few default relations.
>
> > Is the munmap error of concern? It remains upon rebooting / restarting the
> > server.
>
> Seems pretty fishy to me; I don't know what would be causing it.
>
> [ digs in code... ]  One theory is that PGSharedMemoryDetach is getting
> called more than once, but I'm not sure how that would happen.  Can you
> characterize where this happens more precisely?  What nondefault settings
> have you got in postgresql.conf?

Hm. Could that be from the DSM code?

- Andres

Andres Freund <andres@anarazel.de> writes:
> On 2016-10-10 18:21:48 -0400, Tom Lane wrote:
>> Chris Richards <chris@infinite.io> writes:
>>> LOG:  munmap(0x7fff80000000) failed: Invalid argument

>> [ digs in code... ]  One theory is that PGSharedMemoryDetach is getting
>> called more than once, but I'm not sure how that would happen.  Can you
>> characterize where this happens more precisely?  What nondefault settings
>> have you got in postgresql.conf?

> Hm. Could that be from the DSM code?

That particular error message spelling only appears in sysv_shmem.c,
so it's not directly DSM's fault.

The comments around PGSharedMemoryDetach strongly suggest that it ought
to be a no-op if called twice, which it originally was but fails to be
since the addition of the munmap call.   So I'm *very* strongly tempted
to add "AnonymousShmem = NULL;" there and in IpcMemoryDetach.  But
it's not evident why we've not seen this behavior many times already,
so I'd kind of like to find out what's different about Chris's use-case
before assuming that that will fix it.

            regards, tom lane

Chris Richards <chris@infinite.io> writes:
> Oh the email spam :P

No problem.  I'm starting to think that this isn't actually a Postgres
bug, but rather something funny with your kernel.  The code in sysv_shmem
is certainly as fragile as can be, but I'm darned if I can find an
existing code path that would actually break it --- and the lack of other
complaints is pointing towards something funny with your system, too.

First off, does changing "huge_pages" in postgresql.conf alter the
behavior?  (It defaults to "try", see what happens with "on" or "off".)

Second, I notice that you've got

> Hugepagesize:    1048576 kB

which is just enormous.  PG is only going to request circa 140MB given
the settings you mentioned.  We've seen reports of kernel bugs that cause
mmap() to fail for requests that aren't a multiple of the hugepage size,
but I've not heard that mmap() might succeed and then munmap() fail.
That seems like what's happening to you though.

            regards, tom lane