Thread: Upgrade from 9.4 -> 9.5, FreeBSD 10.2-STABLE, fails on initdb

$ initdb -D data-default
The files belonging to this database system will be owned by user "pgsql".
This user must also own the server process.

The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory data-default ... ok
creating subdirectories ... ok
selecting default max_connections ... 10
selecting default shared_buffers ... 400kB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
creating template1 database in data-default/base/1 ... FATAL:  could not create semaphores: Invalid argument
DETAIL:  Failed system call was semget(2, 17, 03600).
child process exited with exit code 1
initdb: removing contents of data directory "data-default"
$
$ sysctl -a|grep semm
kern.ipc.semmsl: 512
kern.ipc.semmnu: 256
kern.ipc.semmns: 512
kern.ipc.semmni: 256

The system is running 9.4 just fine and the kernel configuration requirements shouldn't have changed for semaphores.... should they?

On 02/04/2016 10:02 AM, Karl Denninger wrote:
> $ initdb -D data-default
> The files belonging to this database system will be owned by user "pgsql".
> This user must also own the server process.
>
> The database cluster will be initialized with locale "C".
> The default database encoding has accordingly been set to "SQL_ASCII".
> The default text search configuration will be set to "english".
>
> Data page checksums are disabled.
>
> fixing permissions on existing directory data-default ... ok
> creating subdirectories ... ok
> selecting default max_connections ... 10
> selecting default shared_buffers ... 400kB
> selecting dynamic shared memory implementation ... posix
> creating configuration files ... ok
> creating template1 database in data-default/base/1 ... FATAL:  could not
> create semaphores: Invalid argument
> DETAIL:  Failed system call was semget(2, 17, 03600).
> child process exited with exit code 1
> initdb: removing contents of data directory "data-default"
> $
> $ sysctl -a|grep semm
> kern.ipc.semmsl: 512
> kern.ipc.semmnu: 256
> kern.ipc.semmns: 512
> kern.ipc.semmni: 256
>
> The system is running 9.4 just fine and the kernel configuration
> requirements shouldn't have changed for semaphores.... should they?


Where did the 9.5 version come from and was it the same source as the
9.4 version?

>
> --
> Karl Denninger
> karl@denninger.net <mailto:karl@denninger.net>
> /The Market Ticker/
> /[S/MIME encrypted email preferred]/


--
Adrian Klaver
adrian.klaver@aklaver.com

On 02/04/2016 10:02 AM, Karl Denninger wrote:
> $ initdb -D data-default
> The files belonging to this database system will be owned by user "pgsql".
> This user must also own the server process.
>
> The database cluster will be initialized with locale "C".
> The default database encoding has accordingly been set to "SQL_ASCII".
> The default text search configuration will be set to "english".
>
> Data page checksums are disabled.
>
> fixing permissions on existing directory data-default ... ok
> creating subdirectories ... ok
> selecting default max_connections ... 10
> selecting default shared_buffers ... 400kB
> selecting dynamic shared memory implementation ... posix
> creating configuration files ... ok
> creating template1 database in data-default/base/1 ... FATAL:  could not
> create semaphores: Invalid argument
> DETAIL:  Failed system call was semget(2, 17, 03600).
> child process exited with exit code 1
> initdb: removing contents of data directory "data-default"
> $
> $ sysctl -a|grep semm
> kern.ipc.semmsl: 512
> kern.ipc.semmnu: 256
> kern.ipc.semmns: 512
> kern.ipc.semmni: 256
>
> The system is running 9.4 just fine and the kernel configuration
> requirements shouldn't have changed for semaphores.... should they?

If it helps, the hint from the source code:


http://git.postgresql.org/gitweb/?p=postgresql.git;a=blob;f=src/backend/port/sysv_sema.c;h=f6f15169200a03e9da46ae348994f04297d22017;hb=HEAD

"This error does *not* mean that you have run out of disk space. It
occurs when either the system limit for the maximum number of semaphore
sets (SEMMNI), or the system wide maximum number of semaphores (SEMMNS),
would be exceeded.  You need to raise the respective kernel parameter.
Alternatively, reduce PostgreSQL's consumption of semaphores by reducing
its max_connections parameter.
The PostgreSQL documentation contains more information about configuring
your system for PostgreSQL."


>
> --
> Karl Denninger
> karl@denninger.net <mailto:karl@denninger.net>
> /The Market Ticker/
> /[S/MIME encrypted email preferred]/


--
Adrian Klaver
adrian.klaver@aklaver.com

Karl Denninger <karl@denninger.net> writes:
> $ initdb -D data-default
> ...
> creating template1 database in data-default/base/1 ... FATAL:  could not
> create semaphores: Invalid argument
> DETAIL:  Failed system call was semget(2, 17, 03600).

Hmm.  On my Linux box, "man semget" says EINVAL means

       EINVAL nsems  is less than 0 or greater than the limit on the number of
              semaphores per semaphore set (SEMMSL), or a semaphore set corre-
              sponding  to  key  already  exists, and nsems is larger than the
              number of semaphores in that set.

which agrees with the POSIX spec.  Is FreeBSD the same?

Proceeding on the assumption that it is ...

17 is the same nsems value we've been using for donkey's years, so the
SEMMSL aspect of this seems unlikely to apply; what presumably is
happening is a collision with an existing semaphore's key.  Our code is
prepared for that, but it expects a different error code in such cases,
either EEXIST or EACCES:

        /*
         * Fail quietly if error indicates a collision with existing set. One
         * would expect EEXIST, given that we said IPC_EXCL, but perhaps we
         * could get a permission violation instead?  Also, EIDRM might occur
         * if an old set is slated for destruction but not gone yet.
         */

It sounds like your kernel is returning EINVAL in preference to any of
those codes, which would be pretty broken.  I do not want to make our code
treat EINVAL as meaning we should retry with a different key, because if
the problem is indeed the SEMMSL limit, we'd be in an infinite loop.

You can probably get past this for the moment if you can remove the
semaphore set with key 2, but I'd advise filing a FreeBSD kernel
bug about their choice of errno.

            regards, tom lane

Karl Denninger <karl@denninger.net> writes:

$ initdb -D data-default
...
creating template1 database in data-default/base/1 ... FATAL:  could not
create semaphores: Invalid argument
DETAIL:  Failed system call was semget(2, 17, 03600).

Hmm.  On my Linux box, "man semget" says EINVAL means
      EINVAL nsems  is less than 0 or greater than the limit on the number of             semaphores per semaphore set (SEMMSL), or a semaphore set corre-             sponding  to  key  already  exists, and nsems is larger than the             number of semaphores in that set.

which agrees with the POSIX spec.  Is FreeBSD the same?

Proceeding on the assumption that it is ...

17 is the same nsems value we've been using for donkey's years, so the
SEMMSL aspect of this seems unlikely to apply; what presumably is
happening is a collision with an existing semaphore's key.  Our code is
prepared for that, but it expects a different error code in such cases,
either EEXIST or EACCES:
       /*        * Fail quietly if error indicates a collision with existing set. One        * would expect EEXIST, given that we said IPC_EXCL, but perhaps we        * could get a permission violation instead?  Also, EIDRM might occur        * if an old set is slated for destruction but not gone yet.        */

It sounds like your kernel is returning EINVAL in preference to any of
those codes, which would be pretty broken.  I do not want to make our code
treat EINVAL as meaning we should retry with a different key, because if
the problem is indeed the SEMMSL limit, we'd be in an infinite loop.

You can probably get past this for the moment if you can remove the
semaphore set with key 2, but I'd advise filing a FreeBSD kernel
bug about their choice of errno.
		regards, tom lane

That sounds like it well may be the problem....

s        65564            2 --rw-rw-rw- www      www      www      www                 3 12:29:14  7:56:04


Oh look, the web server process has a semaphore set out with a "2" key....

I've filed a bug report.  Thanks.

I wrote:
> Karl Denninger <karl@denninger.net> writes:
>> $ initdb -D data-default
>> ...
>> creating template1 database in data-default/base/1 ... FATAL:  could not
>> create semaphores: Invalid argument
>> DETAIL:  Failed system call was semget(2, 17, 03600).

> Hmm.  On my Linux box, "man semget" says EINVAL means

>        EINVAL nsems  is less than 0 or greater than the limit on the number of
>               semaphores per semaphore set (SEMMSL), or a semaphore set corre-
>               sponding  to  key  already  exists, and nsems is larger than the
>               number of semaphores in that set.

> which agrees with the POSIX spec.  Is FreeBSD the same?

BTW, looking at the code, I see that during initdb we would have tried
semaphore key 1 before 2.  So presumably, on key 1 we got an error code
that we recognized as meaning "semaphore set already exists", but then on
key 2 we got EINVAL instead.  That makes this even more curious.  I'd
be interested to see what "ipcs -s" says, if you have that command.
(You might need to run it as root to be sure it will show all sempaphores.)

            regards, tom lane

I wrote:

Karl Denninger <karl@denninger.net> writes:

$ initdb -D data-default
...
creating template1 database in data-default/base/1 ... FATAL:  could not
create semaphores: Invalid argument
DETAIL:  Failed system call was semget(2, 17, 03600).

Hmm.  On my Linux box, "man semget" says EINVAL means

       EINVAL nsems  is less than 0 or greater than the limit on the number of             semaphores per semaphore set (SEMMSL), or a semaphore set corre-             sponding  to  key  already  exists, and nsems is larger than the             number of semaphores in that set.

which agrees with the POSIX spec.  Is FreeBSD the same?

BTW, looking at the code, I see that during initdb we would have tried
semaphore key 1 before 2.  So presumably, on key 1 we got an error code
that we recognized as meaning "semaphore set already exists", but then on
key 2 we got EINVAL instead.  That makes this even more curious.  I'd
be interested to see what "ipcs -s" says, if you have that command.
(You might need to run it as root to be sure it will show all sempaphores.)
		regards, tom lane

There was indeed a "2" key out by the web server process; I shut it down and cleared it and the upgrade is now running....

Also filed a kernel bug with the FreeBSD folks against 10.2-STABLE as the man page says you should have gotten back EEXIST.