Thread: scorpionfly needs more semaphores
Hi, We're seeing occasional failures like this: running bootstrap script ... 2019-09-13 12:11:26.882 PDT [64926] FATAL: could not create semaphores: No space left on device 2019-09-13 12:11:26.882 PDT [64926] DETAIL: Failed system call was semget(5728001, 17, 03600). I think you should switch to using "named" POSIX semaphores by building with USE_NAMED_POSIX_SEMAPHORES (then it'll create a squillion little files under /tmp and mmap() them), or increase the number of SysV semaphores you can create with sysctl[1], or finish writing your operating system[2] so you can switch to "unnamed" POSIX semaphores :-) [1] https://www.postgresql.org/message-id/flat/27582.1546928073%40sss.pgh.pa.us [2] https://github.com/openbsd/src/blob/master/lib/librthread/rthread_sem.c#L112 -- Thomas Munro https://enterprisedb.com
Thomas Munro <thomas.munro@gmail.com> writes:
> We're seeing occasional failures like this:
> running bootstrap script ... 2019-09-13 12:11:26.882 PDT [64926]
> FATAL: could not create semaphores: No space left on device
> 2019-09-13 12:11:26.882 PDT [64926] DETAIL: Failed system call was
> semget(5728001, 17, 03600).
> I think you should switch to using "named" POSIX semaphores by
> building with USE_NAMED_POSIX_SEMAPHORES (then it'll create a
> squillion little files under /tmp and mmap() them), or increase the
> number of SysV semaphores you can create with sysctl[1], or finish
> writing your operating system[2] so you can switch to "unnamed" POSIX
> semaphores :-)
I'd recommend the second option. Since the discussion in [1],
we've fixed our docs for OpenBSD to say
In OpenBSD 3.3 and later, IPC parameters can be adjusted using sysctl,
for example:
# sysctl kern.seminfo.semmni=100
To make these settings persist over reboots, modify /etc/sysctl.conf.
You will usually want to increase kern.seminfo.semmni and
kern.seminfo.semmns, as OpenBSD's default settings for these are
uncomfortably small.
Scorpionfly also seems to be having problems with its git repo breaking on
a regular basis. I have no idea what's up with that.
regards, tom lane
Thus said Tom Lane <tgl@sss.pgh.pa.us> on Wed, 18 Sep 2019 00:33:19 -0400 > Thomas Munro <thomas.munro@gmail.com> writes: >> We're seeing occasional failures like this: >> running bootstrap script ... 2019-09-13 12:11:26.882 PDT [64926] >> FATAL: could not create semaphores: No space left on device >> 2019-09-13 12:11:26.882 PDT [64926] DETAIL: Failed system call was >> semget(5728001, 17, 03600). > >> I think you should switch to using "named" POSIX semaphores by >> building with USE_NAMED_POSIX_SEMAPHORES (then it'll create a >> squillion little files under /tmp and mmap() them), or increase the >> number of SysV semaphores you can create with sysctl[1], or finish >> writing your operating system[2] so you can switch to "unnamed" POSIX >> semaphores :-) > > I'd recommend the second option. Since the discussion in [1], > we've fixed our docs for OpenBSD to say > > In OpenBSD 3.3 and later, IPC parameters can be adjusted using sysctl, > for example: > # sysctl kern.seminfo.semmni=100 > To make these settings persist over reboots, modify /etc/sysctl.conf. > You will usually want to increase kern.seminfo.semmni and > kern.seminfo.semmns, as OpenBSD's default settings for these are > uncomfortably small. Thanks, Thomas and Tom for reaching out to me. I certainly don't want to recompile my kernel, as I basically run -current OpenBSD via snapshots. That said, I've made the adjustment to the sysctl: $ sysctl | ag kern.seminfo.semmni kern.seminfo.semmni=100 > > Scorpionfly also seems to be having problems with its git repo breaking on > a regular basis. I have no idea what's up with that. That is a mystery to me as well. 9.4 stable seems to be the branch with the most problems: https://buildfarm.postgresql.org/cgi-bin/show_history.pl?nm=scorpionfly&br=REL9_4_STABLE My cronjobs: 0 */6 * * * cd /home/pgbuilder/bin/REL10 && ./run_build.pl --verbose 0 */12 * * * cd /home/pgbuilder/bin/REL10 && ./run_branches.pl --run-all I'm willing to make more tweaks to prevent these false positives, so feel free to continue monitoring to see how things work out over the next several builds. > > regards, tom lane >
On Wed, Sep 18, 2019 at 4:55 PM jungle boogie <jungleboogie0@gmail.com> wrote: > $ sysctl | ag kern.seminfo.semmni > kern.seminfo.semmni=100 It still seems to be happening. Perhaps you need to increase semmns too? > > Scorpionfly also seems to be having problems with its git repo breaking on > > a regular basis. I have no idea what's up with that. > > That is a mystery to me as well. 9.4 stable seems to be the branch with > the most problems: > https://buildfarm.postgresql.org/cgi-bin/show_history.pl?nm=scorpionfly&br=REL9_4_STABLE > > > My cronjobs: > 0 */6 * * * cd /home/pgbuilder/bin/REL10 && ./run_build.pl --verbose > 0 */12 * * * cd /home/pgbuilder/bin/REL10 && ./run_branches.pl --run-all I think you need just the run_branches.pl entry. BTW I'm sorry for my flippant tone about sem_init() earlier, which someone pointed out to me was not great cross-project open source. What I really meant to say was: if you have OpenBSD developer contacts, it would be cool if you could highlight that issue as something that would make the PostgreSQL-on-OpenBSD experience nicer for end users (and I suspect other multi-process software too). On Linux and FreeBSD we now use sem_init() (PREFERRED_SEMAPHORES=UNNAMED_POSIX) so users never need to worry about configuring that kernel resource. On at least AIX, we still have to use SysV, but there the default limits are high enough that (according to our manual) no adjustment is required. Also, as I speculated in that other thread: based on a quick peek at the implementation, you might get better performance on very large busy PostgreSQL clusters from our cache line padded sem_init() array than you do with your more densely packed SysV semas (I could be totally wrong about that, but we saw an effect like that on some other operating system). -- Thomas Munro https://enterprisedb.com
On 2019-09-23 00:29, Thomas Munro wrote: > On Wed, Sep 18, 2019 at 4:55 PM jungle boogie <jungleboogie0@gmail.com> wrote: >> $ sysctl | ag kern.seminfo.semmni >> kern.seminfo.semmni=100 > > It still seems to be happening. Perhaps you need to increase semmns too? I have on my OpenBSD 6.5 /etc/sysctl.conf the following: kern.seminfo.semmni=200 kern.seminfo.semmns=4000 and it seems to work fine. /Mikael
On Mon Sep 23, 2019 at 8:34 PM Mikael Kjellström wrote: > On 2019-09-23 00:29, Thomas Munro wrote: > > On Wed, Sep 18, 2019 at 4:55 PM jungle boogie <jungleboogie0@gmail.com> wrote: > >> $ sysctl | ag kern.seminfo.semmni > >> kern.seminfo.semmni=100 > > > > It still seems to be happening. Perhaps you need to increase semmns too? > > I have on my OpenBSD 6.5 /etc/sysctl.conf the following: > > kern.seminfo.semmni=200 > kern.seminfo.semmns=4000 > > and it seems to work fine. Thanks! I've made these adjustments and we'll see what happens. Thomas, No offense taken with your input from last week. Your additional input this week is welcomed and I'll pass along to some folks. Thank you all for continuing to monitor Postgresql building on OpenBSD. > > /Mikael