Thread: scorpionfly needs more semaphores

scorpionfly needs more semaphores

From
Thomas Munro
Date:
Hi,

We're seeing occasional failures like this:

running bootstrap script ... 2019-09-13 12:11:26.882 PDT [64926]
FATAL:  could not create semaphores: No space left on device
2019-09-13 12:11:26.882 PDT [64926] DETAIL:  Failed system call was
semget(5728001, 17, 03600).

I think you should switch to using "named" POSIX semaphores by
building with USE_NAMED_POSIX_SEMAPHORES (then it'll create a
squillion little files under /tmp and mmap() them), or increase the
number of SysV semaphores you can create with sysctl[1], or finish
writing your operating system[2] so you can switch to "unnamed" POSIX
semaphores :-)

[1] https://www.postgresql.org/message-id/flat/27582.1546928073%40sss.pgh.pa.us
[2] https://github.com/openbsd/src/blob/master/lib/librthread/rthread_sem.c#L112

-- 
Thomas Munro
https://enterprisedb.com



Re: scorpionfly needs more semaphores

From
Tom Lane
Date:
Thomas Munro <thomas.munro@gmail.com> writes:
> We're seeing occasional failures like this:
> running bootstrap script ... 2019-09-13 12:11:26.882 PDT [64926]
> FATAL:  could not create semaphores: No space left on device
> 2019-09-13 12:11:26.882 PDT [64926] DETAIL:  Failed system call was
> semget(5728001, 17, 03600).

> I think you should switch to using "named" POSIX semaphores by
> building with USE_NAMED_POSIX_SEMAPHORES (then it'll create a
> squillion little files under /tmp and mmap() them), or increase the
> number of SysV semaphores you can create with sysctl[1], or finish
> writing your operating system[2] so you can switch to "unnamed" POSIX
> semaphores :-)

I'd recommend the second option.  Since the discussion in [1],
we've fixed our docs for OpenBSD to say

    In OpenBSD 3.3 and later, IPC parameters can be adjusted using sysctl,
    for example:
    # sysctl kern.seminfo.semmni=100
    To make these settings persist over reboots, modify /etc/sysctl.conf.
    You will usually want to increase kern.seminfo.semmni and
    kern.seminfo.semmns, as OpenBSD's default settings for these are
    uncomfortably small.

Scorpionfly also seems to be having problems with its git repo breaking on
a regular basis.  I have no idea what's up with that.

            regards, tom lane



Re: scorpionfly needs more semaphores

From
jungle boogie
Date:
Thus said Tom Lane <tgl@sss.pgh.pa.us> on Wed, 18 Sep 2019 00:33:19 -0400
> Thomas Munro <thomas.munro@gmail.com> writes:
>> We're seeing occasional failures like this:
>> running bootstrap script ... 2019-09-13 12:11:26.882 PDT [64926]
>> FATAL:  could not create semaphores: No space left on device
>> 2019-09-13 12:11:26.882 PDT [64926] DETAIL:  Failed system call was
>> semget(5728001, 17, 03600).
> 
>> I think you should switch to using "named" POSIX semaphores by
>> building with USE_NAMED_POSIX_SEMAPHORES (then it'll create a
>> squillion little files under /tmp and mmap() them), or increase the
>> number of SysV semaphores you can create with sysctl[1], or finish
>> writing your operating system[2] so you can switch to "unnamed" POSIX
>> semaphores :-)
> 
> I'd recommend the second option.  Since the discussion in [1],
> we've fixed our docs for OpenBSD to say
> 
>      In OpenBSD 3.3 and later, IPC parameters can be adjusted using sysctl,
>      for example:
>      # sysctl kern.seminfo.semmni=100
>      To make these settings persist over reboots, modify /etc/sysctl.conf.
>      You will usually want to increase kern.seminfo.semmni and
>      kern.seminfo.semmns, as OpenBSD's default settings for these are
>      uncomfortably small.


Thanks, Thomas and Tom for reaching out to me. I certainly don't want to 
recompile my kernel, as I basically run -current OpenBSD via snapshots.

That said, I've made the adjustment to the sysctl:

$ sysctl | ag kern.seminfo.semmni
kern.seminfo.semmni=100


> 
> Scorpionfly also seems to be having problems with its git repo breaking on
> a regular basis.  I have no idea what's up with that.

That is a mystery to me as well. 9.4 stable seems to be the branch with 
the most problems:
https://buildfarm.postgresql.org/cgi-bin/show_history.pl?nm=scorpionfly&br=REL9_4_STABLE


My cronjobs:
0 */6 * * * cd /home/pgbuilder/bin/REL10 && ./run_build.pl  --verbose
0 */12 * * * cd /home/pgbuilder/bin/REL10 && ./run_branches.pl  --run-all


I'm willing to make more tweaks to prevent these false positives, so 
feel free to continue monitoring to see how things work out over the 
next several builds.

> 
>             regards, tom lane
> 






Re: scorpionfly needs more semaphores

From
Thomas Munro
Date:
On Wed, Sep 18, 2019 at 4:55 PM jungle boogie <jungleboogie0@gmail.com> wrote:
> $ sysctl | ag kern.seminfo.semmni
> kern.seminfo.semmni=100

It still seems to be happening.  Perhaps you need to increase semmns too?

> > Scorpionfly also seems to be having problems with its git repo breaking on
> > a regular basis.  I have no idea what's up with that.
>
> That is a mystery to me as well. 9.4 stable seems to be the branch with
> the most problems:
> https://buildfarm.postgresql.org/cgi-bin/show_history.pl?nm=scorpionfly&br=REL9_4_STABLE
>
>
> My cronjobs:
> 0 */6 * * * cd /home/pgbuilder/bin/REL10 && ./run_build.pl  --verbose
> 0 */12 * * * cd /home/pgbuilder/bin/REL10 && ./run_branches.pl  --run-all

I think you need just the run_branches.pl entry.

BTW I'm sorry for my flippant tone about sem_init() earlier, which
someone pointed out to me was not great cross-project open source.
What I really meant to say was: if you have OpenBSD developer
contacts, it would be cool if you could highlight that issue as
something that would make the PostgreSQL-on-OpenBSD experience nicer
for end users (and I suspect other multi-process software too).  On
Linux and FreeBSD we now use sem_init()
(PREFERRED_SEMAPHORES=UNNAMED_POSIX) so users never need to worry
about configuring that kernel resource.  On at least AIX, we still
have to use SysV, but there the default limits are high enough that
(according to our manual) no adjustment is required.  Also, as I
speculated in that other thread: based on a quick peek at the
implementation, you might get better performance on very large busy
PostgreSQL clusters from our cache line padded sem_init() array than
you do with your more densely packed SysV semas (I could be totally
wrong about that, but we saw an effect like that on some other
operating system).

-- 
Thomas Munro
https://enterprisedb.com



Re: scorpionfly needs more semaphores

From
Mikael Kjellström
Date:
On 2019-09-23 00:29, Thomas Munro wrote:
> On Wed, Sep 18, 2019 at 4:55 PM jungle boogie <jungleboogie0@gmail.com> wrote:
>> $ sysctl | ag kern.seminfo.semmni
>> kern.seminfo.semmni=100
> 
> It still seems to be happening.  Perhaps you need to increase semmns too?

I have on my OpenBSD 6.5 /etc/sysctl.conf the following:

kern.seminfo.semmni=200
kern.seminfo.semmns=4000

and it seems to work fine.

/Mikael



Re: scorpionfly needs more semaphores

From
"Jungle Boogie"
Date:
On Mon Sep 23, 2019 at 8:34 PM Mikael Kjellström wrote:
> On 2019-09-23 00:29, Thomas Munro wrote:
> > On Wed, Sep 18, 2019 at 4:55 PM jungle boogie <jungleboogie0@gmail.com> wrote:
> >> $ sysctl | ag kern.seminfo.semmni
> >> kern.seminfo.semmni=100
> >
> > It still seems to be happening.  Perhaps you need to increase semmns too?
>
> I have on my OpenBSD 6.5 /etc/sysctl.conf the following:
>
> kern.seminfo.semmni=200
> kern.seminfo.semmns=4000
>
> and it seems to work fine.

Thanks! I've made these adjustments and we'll see what happens.

Thomas,

No offense taken with your input from last week. Your additional input this week
is welcomed and I'll pass along to some folks.

Thank you all for continuing to monitor Postgresql building on OpenBSD.

>
> /Mikael