Thread: Fresh build on OS X not working (memory)
Mac OS X 10.6.3
Built and installed PG 8.4.3 from source
Trying to start the server, I get "could not create shared memory segment: Invalid argument".
I've bumped my shmmax/shmall and related parameters as suggested in the README for the one-click installer (which I installed and then uninstalled before building from source), and then even higher as suggested by users on IRC. No luck.
This is for a dev laptop (2GB RAM), so I don't care about tuning performance. I just want pg to get up and running at all.
Any suggestions for what to try next are welcome. Relevant details follow:
Slim2:pgsql _postgres$ sysctl -a | grep shmkern.sysv.shmmax: 2147483648kern.sysv.shmmin: 1kern.sysv.shmmni: 32kern.sysv.shmseg: 8kern.sysv.shmall: 2147483648machdep.pmap.hashmax: 12security.mac.posixshm_enforce: 1security.mac.sysvshm_enforce: 1Slim2:pgsql _postgres$ cat data/postgresql.conf | grep -E "max_conn|shared_buff"max_connections = 10 # (change requires restart)# Note: Increasing max_connections costs ~400 bytes of shared memory pershared_buffers = 16MB # min 128kB# max_locks_per_transaction * (max_connections + max_prepared_transactions)Slim2:pgsql _postgres$ /usr/local/pgsql/bin/pg_ctl -D /usr/local/pgsql/data startserver startingSlim2:pgsql _postgres$ FATAL: could not create shared memory segment: Invalid argumentDETAIL: Failed system call was shmget(key=5432001, size=18366464, 03600).HINT: This error usually means that PostgreSQL's request for a shared memory segment exceeded your kernel's SHMMAX parameter. You can either reduce the request size or reconfigure the kernel with larger SHMMAX. To reduce the request size (currently 18366464 bytes), reduce PostgreSQL's shared_buffers parameter (currently 2048) and/or its max_connections parameter (currently 13).If the request size is already small, it's possible that it is less than your kernel's SHMMIN parameter, in which case raising the request size or reconfiguring SHMMIN is called for.The PostgreSQL documentation contains more information about shared memory configuration.
Gavin Kistner <phrogz@mac.com> writes: > Mac OS X 10.6.3 > Built and installed PG 8.4.3 from source > Trying to start the server, I get "could not create shared memory segment: Invalid argument". OSX is *notoriously* unfriendly about configuring shm parameters. You look at it sidewise, it doesn't work, and generally won't tell you why it's refusing to talk to you. We document a few of the known constraints in our manual, but I'm sure there are more. > Slim2:pgsql _postgres$ sysctl -a | grep shm > kern.sysv.shmmax: 2147483648 > kern.sysv.shmmin: 1 > kern.sysv.shmmni: 32 > kern.sysv.shmseg: 8 > kern.sysv.shmall: 2147483648 Hm. That shmall setting is not sane at all (remember the units are pages). I wonder whether you've forced some internal overflow inside the kernel. For comparison, I have these settings on my laptop: pro:~ tgl$ cat /etc/sysctl.conf kern.sysv.shmmax=33554432 kern.sysv.shmmin=1 kern.sysv.shmmni=32 kern.sysv.shmseg=8 kern.sysv.shmall=8192 and PG starts fine for me. regards, tom lane
On Apr 29, 2010, at 11:42 PM, Tom Lane wrote: > For comparison, I have these settings on my laptop: > > pro:~ tgl$ cat /etc/sysctl.conf > kern.sysv.shmmax=33554432 > kern.sysv.shmmin=1 > kern.sysv.shmmni=32 > kern.sysv.shmseg=8 > kern.sysv.shmall=8192 > > and PG starts fine for me. I'm not sure what magic you have in there, but this incantation copied verbatim worked for me. Thanks! The first sysctl.conf I tried that didn't work from from the one-click installer's README. Do you see anything about thefollowing settings that jumps out at you as bad? Is it notable that yours don't set the maxproc*? kern.sysv.shmmax=1610612736 kern.sysv.shmall=393216 kern.sysv.shmmin=1 kern.sysv.shmmni=32 kern.sysv.shmseg=8 kern.maxprocperuid=512 kern.maxproc=2048
Gavin Kistner <phrogz@mac.com> writes: > On Apr 29, 2010, at 11:42 PM, Tom Lane wrote: >> For comparison, I have these settings on my laptop: >> pro:~ tgl$ cat /etc/sysctl.conf >> kern.sysv.shmmax=33554432 >> kern.sysv.shmmin=1 >> kern.sysv.shmmni=32 >> kern.sysv.shmseg=8 >> kern.sysv.shmall=8192 >> >> and PG starts fine for me. > I'm not sure what magic you have in there, but this incantation copied verbatim worked for me. Thanks! > The first sysctl.conf I tried that didn't work from from the one-click installer's README. Do you see anything about thefollowing settings that jumps out at you as bad? Is it notable that yours don't set the maxproc*? > kern.sysv.shmmax=1610612736 > kern.sysv.shmall=393216 > kern.sysv.shmmin=1 > kern.sysv.shmmni=32 > kern.sysv.shmseg=8 > kern.maxprocperuid=512 > kern.maxproc=2048 I have no idea why EDB recommend changing the maxproc settings, but I doubt that's related to shared memory. I see that they have shmmax equal to exactly 4096 times shmall, so that's good, but there must be some other OSX peculiarity that this is tripping over. Maybe it's too large ... do you actually have 1.6GB of RAM available that could reasonably be dedicated to PG shared memory? Another possibility is that maybe only exact powers of 2 work well. I'm just guessing though. regards, tom lane
I wrote: > I have no idea why EDB recommend changing the maxproc settings, but > I doubt that's related to shared memory. I see that they have shmmax > equal to exactly 4096 times shmall, so that's good, but there must be > some other OSX peculiarity that this is tripping over. Maybe it's too > large ... do you actually have 1.6GB of RAM available that could > reasonably be dedicated to PG shared memory? Another possibility is > that maybe only exact powers of 2 work well. I'm just guessing though. I poked around in the OS X kernel sources (let's hear it for open source) and found that there really isn't a lot of special magic around the shm control parameters anymore: as of OS X 10.6, they're all 64 bits and the comparisons are entirely straightforward. If you can get the values into the system, which you did since sysctl was reporting them, they should work as-expected. So why were you getting an EINVAL failure? The only theory that seems to hold water after looking at the kernel source code is that there was a pre-existing shm segment with the same key, and you got to this bit in shmget_existing(): if (uap->size && uap->size > shmseg->u.shm_segsz) return EINVAL; if ((uap->shmflg & (IPC_CREAT | IPC_EXCL)) == (IPC_CREAT | IPC_EXCL)) return EEXIST; IOW, if there's an existing shm segment of the same key but it's too small, you get EINVAL. Not EEXIST, which is what our code in InternalIpcMemoryCreate is expecting to see for a collision. So our code fails to recognize the case as a collision and spits out a misleading error message. If memory serves, we've seen this issue before [ ... digs in archives ... ] ah-hah: http://archives.postgresql.org/pgsql-hackers/2008-10/msg00896.php I argued in that message that returning EINVAL when EEXIST would apply is a kernel bug, and I still think that. But what seems likely at this point is that many BSD-derived kernels behave this way, and we're never gonna get 'em all fixed. So it would behoove us to work around it. Does the theory of a pre-existing smaller shmem segment make sense from your end? In particular, had you previously had another Postgres server running on that machine, and perhaps killed it ungracefully? If this theory is correct, the issue was "fixed" as a result of rebooting (thus making the old segment go away), not as a result of any changes of the shm parameters. OTOH I'd have expected you to have to reboot multiple times while experimenting with the shm parameters, so I'm not entirely convinced I've hit on the right explanation. regards, tom lane
I wrote: > ... OTOH I'd have expected you to have to reboot multiple > times while experimenting with the shm parameters, so I'm not entirely > convinced I've hit on the right explanation. Oh ... were you using sysctl -w rather than rebooting to adjust the shmmem parameters? I realized after experimenting that 10.6 lets you do that. It didn't use to be possible to change these parameters in OS X without rebooting --- I wonder when they fixed that. regards, tom lane
On Apr 30, 2010, at 3:35 PM, Tom Lane wrote: > Does the theory of a pre-existing smaller shmem segment make sense from > your end? In particular, had you previously had another Postgres server > running on that machine, and perhaps killed it ungracefully? If this > theory is correct, the issue was "fixed" as a result of rebooting (thus > making the old segment go away), not as a result of any changes of the > shm parameters. OTOH I'd have expected you to have to reboot multiple > times while experimenting with the shm parameters, so I'm not entirely > convinced I've hit on the right explanation. It seems probably that you have hit the nail squarely on the head. I had installed from the one-click (which was workingwith the memory changes supplied in the README), and IIRC I tried to kill a few processes in preparation for a manualuninstall before I found the supplied uninstall app. I then compiled and installed from source. When that failed asdescribed earlier, all changes I made to the settings were (as you guessed) through the sysctl -w option without a reboot.The first reboot I had was after I replaced my sysctl.conf wholesale with yours, and all was well. Nice sleuthing. If I didn't have so many things going on at the moment I'd revert to the supplied settings and test to ensureit works and provide a confirmation. I'll try to remember to do that later. Thanks again for your help and good information.
Gavin Kistner <phrogz@mac.com> writes: > On Apr 30, 2010, at 3:35 PM, Tom Lane wrote: >> Does the theory of a pre-existing smaller shmem segment make sense from >> your end? In particular, had you previously had another Postgres server >> running on that machine, and perhaps killed it ungracefully? > It seems probably that you have hit the nail squarely on the head. I had installed from the one-click (which was workingwith the memory changes supplied in the README), and IIRC I tried to kill a few processes in preparation for a manualuninstall before I found the supplied uninstall app. I then compiled and installed from source. When that failed asdescribed earlier, all changes I made to the settings were (as you guessed) through the sysctl -w option without a reboot.The first reboot I had was after I replaced my sysctl.conf wholesale with yours, and all was well. Yeah, I was able to reproduce the problem locally by kill -9'ing the postmaster and then enlarging max_connections in postgresql.conf before trying to restart. In your case it might've been caused by the new version needing a bit more shared memory rather than by any change in parameter settings, but the basic issue seems pretty clear now. I've applied a patch to our CVS to make PG behave more sanely in this case. It does do the right thing if it recognizes that the shmget failure is due to a pre-existing segment, but we weren't being sufficiently careful about distinguishing that from a bad size request. Thanks for the report! regards, tom lane
On Thu, Apr 29, 2010 at 11:42 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > Gavin Kistner <phrogz@mac.com> writes: >> Mac OS X 10.6.3 >> Built and installed PG 8.4.3 from source >> Trying to start the server, I get "could not create shared memory segment: Invalid argument". > > OSX is *notoriously* unfriendly about configuring shm parameters. > You look at it sidewise, it doesn't work, and generally won't tell > you why it's refusing to talk to you. We document a few of the known > constraints in our manual, but I'm sure there are more. The fact that they seem to change the needed magical incantations every point release doesn't help things either.
Scott Marlowe <scott.marlowe@gmail.com> writes: > On Thu, Apr 29, 2010 at 11:42 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote: >> OSX is *notoriously* unfriendly about configuring shm parameters. >> You look at it sidewise, it doesn't work, and generally won't tell >> you why it's refusing to talk to you. �We document a few of the known >> constraints in our manual, but I'm sure there are more. > The fact that they seem to change the needed magical incantations > every point release doesn't help things either. Actually, it appears they've finally reached a point of reasonable sanity in 10.6. It's still true that they reject non-page-multiple SHMMAX, instead of rounding to the next page boundary like everybody else does; but the old kluges whereby you could only set the values by waving a dead chicken in a specific fashion during boot seem to be mostly gone. I wasn't aware of this myself, not having experimented recently. regards, tom lane