Thread: shared memory on OS X - 7.4beta4
I get this when I try to start up a freshly compiled beta4 on OS X 10.2.6: FATAL: could not create shared memory segment: Invalid argument I saw a previous thread on this for beta2. It sounded like this was a bug that was to be fixed in beta3, but I'm still having this problem with beta4. Thanks, John DeSoi, Ph.D.
I don't know what this error is particularly, but just to say that 7.4beta5 compiles and starts up fine on my OSX 10.2.8 machine Cheers Adam > I get this when I try to start up a freshly compiled beta4 on OS X > 10.2.6: > > FATAL: could not create shared memory segment: Invalid argument > > I saw a previous thread on this for beta2. It sounded like this was a > bug that was to be fixed in beta3, but I'm still having this problem > with beta4. > > Thanks, > > John DeSoi, Ph.D. > > > ---------------------------(end of broadcast)--------------------------- > TIP 3: if posting/reading through Usenet, please send an appropriate > subscribe-nomail command to majordomo@postgresql.org so that your > message can get through to the mailing list cleanly -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean.
>>>>> "JD" == John DeSoi <jd@icx.net> writes: JD> I get this when I try to start up a freshly compiled beta4 on OS X JD> 10.2.6: JD> FATAL: could not create shared memory segment: Invalid argument How much SHM are you requesting (shared_buffers), and does the system allow for that much? There's some bootup script you can modify in OSX to up that limit if you need. Check google and/or Postgres' techdocs site. -- =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= Vivek Khera, Ph.D. Khera Communications, Inc. Internet: khera@kciLink.com Rockville, MD +1-240-453-8497 AIM: vivekkhera Y!: vivek_khera http://www.khera.org/~vivek/
Hi all, Just wanted to know how postgres handles semaphores. Was hoping that i can can use the locks defined in lwlock.h and lwlock.c . If i create a new lock and then use LockAcquire and LockRelease when I want a process to start and stop will this work? Nailah
Nailah Ogeer wrote: > Hi all, > Just wanted to know how postgres handles semaphores. Was hoping that i can > can use the locks defined in lwlock.h and lwlock.c . If i create a new > lock and then use LockAcquire and > LockRelease when I want a process to start and stop will this work? Uh, well, we usually try to get a lock via a test-and-set assembly language call into shared memory, and if we fail, we then get a semaphore and wait for someone to wake us up. -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania 19073
On Friday, October 24, 2003, at 11:26 AM, Vivek Khera wrote: > How much SHM are you requesting (shared_buffers), and does the system > allow for that much? There's some bootup script you can modify in OSX > to up that limit if you need. Check google and/or Postgres' techdocs > site. I did a default configure and did not change anything in the .conf file. It requests 10444800 bytes. But even if I drop max_connections down to 10, it still won't start. Yes, I can modify SHMMAX for the operating system to make it work but I'm trying to understand why I need to do this when it was not necessary under 7.3 or an earlier 7.4 development build (pre beta 1). I was hoping it is a bug, because I think it hurts the adoption of Postgresql if I have to worry about a lot of low level OS settings just to setup a simple database. Best, John DeSoi, Ph.D.
On Friday, October 24, 2003, at 10:19 AM, Adam Witney wrote:
> I don't know what this error is particularly, but just to say that
> 7.4beta5
> compiles and starts up fine on my OSX 10.2.8 machine
OK, I compiled beta 5 and put my rest SHMMAX back to the default
settings as provided by OS X. Still the same error:
FATAL: could not create shared memory segment: Invalid argument
DETAIL: Failed system call was shmget(key=5432001, size=10444800,
03600).
HINT: This error usually means that PostgreSQL's request for a shared
memory segment exceeded your kernel's SHMMAX parameter. You can either
reduce the request size or reconfigure the kernel with larger SHMMAX.
To reduce the request size (currently 10444800 bytes), reduce
PostgreSQL's shared_buffers parameter (currently 1000) and/or its
max_connections parameter (currently 100).
If the request size is already small, it's possible that it is less
than your kernel's SHMMIN parameter, in which case raising the request
size or reconfiguring SHMMIN is called for.
The PostgreSQL documentation contains more information about shared
memory configuration.
Here is how I configured it:
./configure --prefix=/Unix/postgresql --enable-locale
--without-readline --with-includes=/sw/include --with-libraries=/sw/lib
--with-openssl=/sw/lib
Next I upgraded to 10.2.8. Same issue.
So you have not modified your memory settings in
/System/Library/StartupItems/SystemTuning/SystemTuning and you don't
get this error?
I'm using a PowerBook G4 with 512M of memory.
Thanks,
John DeSoi, Ph.D.
On 26/10/03 1:01 am, "John DeSoi" <jd@icx.net> wrote:
>
> On Friday, October 24, 2003, at 10:19 AM, Adam Witney wrote:
>
>> I don't know what this error is particularly, but just to say that
>> 7.4beta5
>> compiles and starts up fine on my OSX 10.2.8 machine
>
> OK, I compiled beta 5 and put my rest SHMMAX back to the default
> settings as provided by OS X. Still the same error:
>
> FATAL: could not create shared memory segment: Invalid argument
> DETAIL: Failed system call was shmget(key=5432001, size=10444800,
> 03600).
> HINT: This error usually means that PostgreSQL's request for a shared
> memory segment exceeded your kernel's SHMMAX parameter. You can either
> reduce the request size or reconfigure the kernel with larger SHMMAX.
> To reduce the request size (currently 10444800 bytes), reduce
> PostgreSQL's shared_buffers parameter (currently 1000) and/or its
> max_connections parameter (currently 100).
> If the request size is already small, it's possible that it is less
> than your kernel's SHMMIN parameter, in which case raising the request
> size or reconfiguring SHMMIN is called for.
> The PostgreSQL documentation contains more information about shared
> memory configuration.
>
> Here is how I configured it:
>
> ./configure --prefix=/Unix/postgresql --enable-locale
> --without-readline --with-includes=/sw/include --with-libraries=/sw/lib
> --with-openssl=/sw/lib
>
>
> Next I upgraded to 10.2.8. Same issue.
>
> So you have not modified your memory settings in
> /System/Library/StartupItems/SystemTuning/SystemTuning and you don't
> get this error?
Ah, yes I did change these... Mine are set to
sysctl -w kern.sysv.shmmax=4194304
sysctl -w kern.sysv.shmmin=1
sysctl -w kern.sysv.shmmni=32
sysctl -w kern.sysv.shmseg=8
sysctl -w kern.sysv.shmall=1024
> I'm using a PowerBook G4 with 512M of memory.
Same as me
Do you have many other applications running? Or an old version of
PostgreSQL... I found when running the regression tests I had to shut down
my 7.3.2 installation
Cheers
adam
adam
--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.
Adam, On Sunday, October 26, 2003, at 10:16 AM, Adam Witney wrote: > Ah, yes I did change these... Mine are set to > > sysctl -w kern.sysv.shmmax=4194304 > sysctl -w kern.sysv.shmmin=1 > sysctl -w kern.sysv.shmmni=32 > sysctl -w kern.sysv.shmseg=8 > sysctl -w kern.sysv.shmall=1024 OK, this is what I was after. I did not have to change any of this for 7.3, but now 7.4 does not work unless on a standard OS X installation unless the user modifies these system settings. I can live with it, but I was hoping it might be a bug that could be resolved before the final 7.4. Every person that installs on OS X is going to run into this. Tracking it down and understanding what you have to change is not simple for us OS X users that are not familiar with Unix :). Best, John DeSoi, Ph.D.
On Sun, Oct 26, 2003 at 04:41:16PM -0500, John DeSoi wrote: > On Sunday, October 26, 2003, at 10:16 AM, Adam Witney wrote: > > >Ah, yes I did change these... Mine are set to > > > > sysctl -w kern.sysv.shmmax=4194304 > > sysctl -w kern.sysv.shmmin=1 > > sysctl -w kern.sysv.shmmni=32 > > sysctl -w kern.sysv.shmseg=8 > > sysctl -w kern.sysv.shmall=1024 > > > OK, this is what I was after. I did not have to change any of this for > 7.3, but now 7.4 does not work unless on a standard OS X installation > unless the user modifies these system settings. Huh, this is strange. IIRC during the installation process, initdb is supposed to determine which values actually work; those are affected by the shmmax operating system mostly, and the idea is to enable the installation process to pick more reasonable values out-of-the-box for memory usage that will work on your system. How did you install Postgres, and what values did initdb pick for shared_buffers and max_connections? -- Alvaro Herrera (<alvherre[a]dcc.uchile.cl>) "Puedes vivir solo una vez, pero si lo haces bien, una vez es suficiente"
On 26/10/03 9:41 pm, "John DeSoi" <jd@icx.net> wrote: > Adam, > > On Sunday, October 26, 2003, at 10:16 AM, Adam Witney wrote: > >> Ah, yes I did change these... Mine are set to >> >> sysctl -w kern.sysv.shmmax=4194304 >> sysctl -w kern.sysv.shmmin=1 >> sysctl -w kern.sysv.shmmni=32 >> sysctl -w kern.sysv.shmseg=8 >> sysctl -w kern.sysv.shmall=1024 > > > OK, this is what I was after. I did not have to change any of this for > 7.3, but now 7.4 does not work unless on a standard OS X installation > unless the user modifies these system settings. I can live with it, but > I was hoping it might be a bug that could be resolved before the final > 7.4. Every person that installs on OS X is going to run into this. > Tracking it down and understanding what you have to change is not > simple for us OS X users that are not familiar with Unix :). Well I had to change this for 7.3 to run and I never set it back to try 7.4 with the defaults But I think the 7.4 installation process checks for the best values of max_connections and shared_buffers that for the current settings shmmax settings But someone else on the list is probably better to answer the technical details of this adam
On Sunday, October 26, 2003, at 05:51 PM, Alvaro Herrera wrote:
> Huh, this is strange. IIRC during the installation process, initdb is
> supposed to determine which values actually work; those are affected by
> the shmmax operating system mostly, and the idea is to enable the
> installation process to pick more reasonable values out-of-the-box for
> memory usage that will work on your system.
>
> How did you install Postgres, and what values did initdb pick for
> shared_buffers and max_connections?
Here were my steps to install and attempt to run it:
Install (clean install to a new directory)
./configure --prefix=/Unix/postgresql --enable-locale
--without-readline --with-includes=/sw/include --with-libraries=/sw/lib
--with-openssl=/sw/lib
make
make install
Initdb:
initdb -D /data/pgtest --no-locale
Startup:
pg_ctl start -D /data/pgtest -l /data/pgtest/logfile -o '-i -p 5432'
the postgresql.conf has
max_connections = 100
shared_buffers = 1000
And here are are the default settings for OS X 10.2.6:
sysctl -w kern.sysv.shmmax=4194304
sysctl -w kern.sysv.shmmin=1
sysctl -w kern.sysv.shmmni=32
sysctl -w kern.sysv.shmseg=8
sysctl -w kern.sysv.shmall=1024
On Sun, Oct 26, 2003 at 10:17:50PM -0500, John DeSoi wrote: > On Sunday, October 26, 2003, at 05:51 PM, Alvaro Herrera wrote: > > >How did you install Postgres, and what values did initdb pick for > >shared_buffers and max_connections? > > the postgresql.conf has > max_connections = 100 > shared_buffers = 1000 > > And here are are the default settings for OS X 10.2.6: > > sysctl -w kern.sysv.shmmax=4194304 Well, I can only say that if I lower shmmax to that value, initdb picks max_connections = 50 shared_buffers = 300 I remember there was a subtle bug in earlier versions corrected after beta4. Can you try beta5 and see if it works for you? But with beta4 I get 400/20 instead of 300/50, so this is probably not the same problem. 1000/100 are actually the largest probed values, so maybe initdb is failing to probe or the postgresql.conf is not written correctly. Does the initdb output actually say that it is going to use 1000/100? And is the generated postgresql.conf by that initdb run the file that gets actually used? Maybe this is useless anyway, because I'm on Linux. Someone with an actual OS X system could probably be a lot more helpful. -- Alvaro Herrera (<alvherre[@]dcc.uchile.cl>) "Cuando no hay humildad las personas se degradan" (A. Christie)
On Sunday, October 26, 2003, at 11:13 PM, Alvaro Herrera Munoz wrote:
>
> I remember there was a subtle bug in earlier versions corrected after
> beta4. Can you try beta5 and see if it works for you? But with beta4
> I
> get 400/20 instead of 300/50, so this is probably not the same problem.
I mentioned earlier in the thread -- I did try beta5 -- same result.
>
> 1000/100 are actually the largest probed values, so maybe initdb is
> failing to probe or the postgresql.conf is not written correctly. Does
> the initdb output actually say that it is going to use 1000/100? And
> is
> the generated postgresql.conf by that initdb run the file that gets
> actually used?
>
I don't recall seeing the initdb output, but I'm sure the generated
postgresql.conf is the one being used. I lowered max_connections and
saw lower numbers in the failed request when I tried to start the
server.
Best,
John DeSoi, Ph.D.
#-----------------------------------------------------------------------
----
# CONNECTIONS AND AUTHENTICATION
#-----------------------------------------------------------------------
----
# - Connection Settings -
#tcpip_socket = false
max_connections = 100
# note: increasing max_connections costs about 500 bytes of shared
# memory per connection slot, in addition to costs from shared_buffers
# and max_locks_per_transaction.
#superuser_reserved_connections = 2
#port = 5432
#unix_socket_directory = ''
#unix_socket_group = ''
#unix_socket_permissions = 0777 # octal
#virtual_host = '' # what interface to listen on; defaults to any
#rendezvous_name = '' # defaults to the computer name
# - Security & Authentication -
#authentication_timeout = 60 # 1-600, in seconds
#ssl = false
#password_encryption = true
#krb_server_keyfile = ''
#db_user_namespace = false
#-----------------------------------------------------------------------
----
# RESOURCE USAGE (except WAL)
#-----------------------------------------------------------------------
----
# - Memory -
shared_buffers = 1000 # min 16, at least max_connections*2, 8KB each
#sort_mem = 1024 # min 64, size in KB
#vacuum_mem = 8192
On 27/10/03 5:46 am, "John DeSoi" <jd@icx.net> wrote: > > On Sunday, October 26, 2003, at 11:13 PM, Alvaro Herrera Munoz wrote: > >> >> I remember there was a subtle bug in earlier versions corrected after >> beta4. Can you try beta5 and see if it works for you? But with beta4 >> I >> get 400/20 instead of 300/50, so this is probably not the same problem. > > I mentioned earlier in the thread -- I did try beta5 -- same result. > >> >> 1000/100 are actually the largest probed values, so maybe initdb is >> failing to probe or the postgresql.conf is not written correctly. Does >> the initdb output actually say that it is going to use 1000/100? And >> is >> the generated postgresql.conf by that initdb run the file that gets >> actually used? >> > > I don't recall seeing the initdb output, but I'm sure the generated > postgresql.conf is the one being used. I lowered max_connections and > saw lower numbers in the failed request when I tried to start the > server. My current settings are max_connections = 32 shared_buffers = 100 But the settings picked automatically by the installation were max_connections = 50 shared_buffers = 300 But I found this would not always start as I often have lots of applications open (this is my laptop) and I had trouble running both 7.3 and 7.4beta5 at the same time Cheers Adam -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean.
>> And here are are the default settings for OS X 10.2.6: >> >> sysctl -w kern.sysv.shmmax=4194304 > >Well, I can only say that if I lower shmmax to that value, initdb picks >max_connections = 50 >shared_buffers = 300 I just did a fresh install of 7.4b5 on a fresh install of MacOS 10.3, and initdb picked those same values for postgresql.conf. PostgreSQL started up the first time successfully, but then refused to start again even after restarting the computer until I reduced shared_buffers. Now to figure out why sysctl -w kern.sysv.shmmax=<some_reasonable_value> doesn't seem to work on 10.3... Jim Crate
Slightly off topic confirmations... >>> And here are are the default settings for OS X 10.2.6: >>> >>> sysctl -w kern.sysv.shmmax=4194304 Seems to be the default on 10.2.8 as well. 10.1.5 doesn't seem to have a kern.sysv.* section: [broccoli:~] erics% sysctl kern.sysv.shmmax second level name sysv in kern.sysv.shmmax is invalid >> >> Well, I can only say that if I lower shmmax to that value, initdb >> picks >> max_connections = 50 >> shared_buffers = 300 > > I just did a fresh install of 7.4b5 on a fresh install of MacOS 10.3, > and initdb > picked those same values for postgresql.conf. PostgreSQL started up > the first > time successfully, but then refused to start again even after > restarting the > computer until I reduced shared_buffers. I've had cases in 10.1.x where after hammering on the database for a while and having to restart it, there was no longer enough shared memory to start postgres, requiring a restart of the computer. On 10.1, I couldn't even get it to change the shmmax value, necessitating some very low buffer numbers (nearly out of the box for pg 7.2, yes, that bad). eric
>>>>> "JC" == Jim Crate <jcrate@deepskytech.com> writes: JC> Now to figure out why sysctl -w JC> kern.sysv.shmmax=<some_reasonable_value> doesn't seem to work on JC> 10.3... I believe you must do it at boot time, not after. ie, it must be done by one of the startup scripts. -- =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= Vivek Khera, Ph.D. Khera Communications, Inc. Internet: khera@kciLink.com Rockville, MD +1-240-453-8497 AIM: vivekkhera Y!: vivek_khera http://www.khera.org/~vivek/
John DeSoi <jd@icx.net> writes:
> OK, I compiled beta 5 and put my rest SHMMAX back to the default
> settings as provided by OS X. Still the same error:
> FATAL: could not create shared memory segment: Invalid argument
> DETAIL: Failed system call was shmget(key=5432001, size=10444800,
> 03600).
> HINT: This error usually means that PostgreSQL's request for a shared
> memory segment exceeded your kernel's SHMMAX parameter. You can either
> reduce the request size or reconfigure the kernel with larger SHMMAX.
> To reduce the request size (currently 10444800 bytes), reduce
> PostgreSQL's shared_buffers parameter (currently 1000) and/or its
> max_connections parameter (currently 100).
There is something very wrong here. On my OS X machine, initdb selects
non-default shared_buffers and max_connections (300 and 50 respectively,
when using 7.4beta5). Why is that failing to happen on yours? I can
see two possibilities: either initdb is failing to notice that it needs
smaller-than-default values, or it's somehow failing to insert those
values into the constructed postgresql.conf file. You could determine
which is happening by looking at the output of initdb to see what it
reports.
[ thinks for a bit... ] Hmm. You *are* using the postgresql.conf file
that initdb built, aren't you?
regards, tom lane
On Tuesday, October 28, 2003, at 03:13 PM, Tom Lane wrote: > There is something very wrong here. On my OS X machine, initdb selects > non-default shared_buffers and max_connections (300 and 50 > respectively, > when using 7.4beta5). Why is that failing to happen on yours? I can > see two possibilities: either initdb is failing to notice that it needs > smaller-than-default values, or it's somehow failing to insert those > values into the constructed postgresql.conf file. You could determine > which is happening by looking at the output of initdb to see what it > reports. I set the memory parameters back to their defaults, and now: it works. I'm getting 300 and 50 with initdb. I don't recall what the output form initdb was from previous attempts, but the .conf file always had 1000 and 100. > > [ thinks for a bit... ] Hmm. You *are* using the postgresql.conf file > that initdb built, aren't you? I have never changed anything in this file (or replaced it). I did not even know what the shared_buffers parameter was for until I ran into this problem. What version of OS X are you running? I thought I replicated the problem after upgrading to 10.2.8, but now I'm not certain I ran initdb again (I may have just rebuilt beta 5). So possibly it could be a difference between 10.2.6 and 10.2.8. If not, I have no clue. I'll report back if I find anything. Best, John DeSoi, Ph.D.
John DeSoi <jd@icx.net> writes:
> What version of OS X are you running? I thought I replicated the
> problem after upgrading to 10.2.8, but now I'm not certain I ran initdb
> again (I may have just rebuilt beta 5). So possibly it could be a
> difference between 10.2.6 and 10.2.8. If not, I have no clue. I'll
> report back if I find anything.
I'm running 10.2.6 (a pretty fresh install, see prior bellyaching about
hardware problems with my laptop ;-)). IIRC, someone else reported
success with a clean 10.2.8 installation in this thread. I have also
checked PG against a 10.3 beta recently, and so have other people.
It's fairly likely that there are problems with 10.1, if anyone still
uses that, but I have no reason to think that PG 7.4 will fail with
either 10.2.* or 10.3.
regards, tom lane