Thread: Estimating total amount of shared memory required by postmaster
Hello, We've recently come across the task of estimating the size of shared memory required for PostgreSQL to start. This comes from the problem of validating postgresql.conf files (http://archives.postgresql.org/pgsql-hackers/2011-03/msg01831.php), i.e. checking that the server will be able to start with new configuration options without actually performing the restart. Currently, I see a couple of ways to get the estimate: - Use the code from ipci.c to get the total size of the shared memory segment that Postmaster would be allocating with thegiven configuration options (shared_buffers, etc.). This would require getting the actual amount of available shared memorysomehow, which is platform dependent and might not be very reliable. The other downside is that the code would needto be updated if the original estimates in ipci.c changes. - Try to actually allocate the shared memory in a way postmaster does this nowadays, if the process fails - analyze the errorcode to check whether the failure is due to the shmmax or shmmall limits being too low. This would need to be run asa separate process (not postmaster's child) to avoid messing with the postmaster's own shared memory, which means thatthis would be hard to implement as a user-callable stored function. I'm also looking for other ideas. Any suggestions? Thank you, Alexey -- Command Prompt, Inc. http://www.CommandPrompt.com PostgreSQL Replication, Consulting, Custom Development, 24x7 support
Alexey Klyukin <alexk@commandprompt.com> writes:
> We've recently come across the task of estimating the size of shared memory
> required for PostgreSQL to start.
> ...
> - Try to actually allocate the shared memory in a way postmaster does this
> nowadays, if the process fails - analyze the error code to check whether the
> failure is due to the shmmax or shmmall limits being too low. This would
> need to be run as a separate process (not postmaster's child) to avoid
> messing with the postmaster's own shared memory, which means that this would
> be hard to implement as a user-callable stored function.
The results of such a test wouldn't be worth the electrons they're
written on anyway: you're ignoring the likelihood that two instances of
shared memory would overrun the kernel's SHMALL limit, when a single
instance would be fine.
Given that you can't do it in the context of a live installation, just
trying to start the postmaster and seeing if it works (same as initdb
does) seems as good as anything else.
regards, tom lane
Excerpts from Tom Lane's message of jue jun 02 15:49:53 -0400 2011: > The results of such a test wouldn't be worth the electrons they're > written on anyway: you're ignoring the likelihood that two instances of > shared memory would overrun the kernel's SHMALL limit, when a single > instance would be fine. Well, overrunning SHMALL sets a different errno than overrunning SHMMAX, so at least it's possible to notice the problem. > Given that you can't do it in the context of a live installation, just > trying to start the postmaster and seeing if it works (same as initdb > does) seems as good as anything else. That's precisely what this is trying to avoid. -- Álvaro Herrera <alvherre@commandprompt.com> The PostgreSQL Company - Command Prompt, Inc. PostgreSQL Replication, Consulting, Custom Development, 24x7 support
On 02.06.2011 21:58, Alexey Klyukin wrote: > Hello, > > We've recently come across the task of estimating the size of shared memory > required for PostgreSQL to start. This comes from the problem of validating > postgresql.conf files > (http://archives.postgresql.org/pgsql-hackers/2011-03/msg01831.php), i.e. > checking that the server will be able to start with new configuration options > without actually performing the restart. Currently, I see a couple of ways > to get the estimate: > > - Use the code from ipci.c to get the total size of the shared memory segment > that Postmaster would be allocating with the given configuration options > (shared_buffers, etc.). This would require getting the actual amount of > available shared memory somehow, which is platform dependent and might not > be very reliable. The other downside is that the code would need to be > updated if the original estimates in ipci.c changes. > > - Try to actually allocate the shared memory in a way postmaster does this > nowadays, if the process fails - analyze the error code to check whether the > failure is due to the shmmax or shmmall limits being too low. This would > need to be run as a separate process (not postmaster's child) to avoid > messing with the postmaster's own shared memory, which means that this would > be hard to implement as a user-callable stored function. > > I'm also looking for other ideas. Any suggestions? There's a patch floating around to use POSIX shared memory, which doesn't obey shmmax and shmmall limits: http://archives.postgresql.org/message-id/D9EDACF7-53F1-4355-84F8-2E74CD19D22D@themactionfaction.com That would allow us to fly under the radar and avoid the whole issue. If you could review that patch, that would be great. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com
On Jun 2, 2011, at 10:49 PM, Tom Lane wrote: > Alexey Klyukin <alexk@commandprompt.com> writes: >> We've recently come across the task of estimating the size of shared memory >> required for PostgreSQL to start. > >> ... > >> - Try to actually allocate the shared memory in a way postmaster does this >> nowadays, if the process fails - analyze the error code to check whether the >> failure is due to the shmmax or shmmall limits being too low. This would >> need to be run as a separate process (not postmaster's child) to avoid >> messing with the postmaster's own shared memory, which means that this would >> be hard to implement as a user-callable stored function. > > The results of such a test wouldn't be worth the electrons they're > written on anyway: you're ignoring the likelihood that two instances of > shared memory would overrun the kernel's SHMALL limit, when a single > instance would be fine. As Alvaro already pointed out, I'm not ignoring shmall. I think that: - shmmax case is more frequent. - there is a way to detect that shmall is a problem. The other question is what to do when we detect that shmall is a problem. I don't have a good answer for that ATM. > > Given that you can't do it in the context of a live installation, just > trying to start the postmaster and seeing if it works (same as initdb > does) seems as good as anything else. In a context of configuration files validator a postmaster restart is definitely not what we are looking for. Alexey. -- Command Prompt, Inc. http://www.CommandPrompt.com PostgreSQL Replication, Consulting, Custom Development, 24x7 support
Excerpts from Tom Lane's message of jue jun 02 15:49:53 -0400 2011: > Alexey Klyukin <alexk@commandprompt.com> writes: > > - Try to actually allocate the shared memory in a way postmaster does this > > nowadays, if the process fails - analyze the error code to check whether the > > failure is due to the shmmax or shmmall limits being too low. This would > > need to be run as a separate process (not postmaster's child) to avoid > > messing with the postmaster's own shared memory, which means that this would > > be hard to implement as a user-callable stored function. > > The results of such a test wouldn't be worth the electrons they're > written on anyway: you're ignoring the likelihood that two instances of > shared memory would overrun the kernel's SHMALL limit, when a single > instance would be fine. > > Given that you can't do it in the context of a live installation, just > trying to start the postmaster and seeing if it works (same as initdb > does) seems as good as anything else. BTW the other idea we discussed at PGCon was that we could have a postmaster option that would parse the config file and then report the amount of shared memory needed to run with it. If we had that, then it's easy to write a platform-specific shell script or program that verifies that the given number is within the allowed limits. -- Álvaro Herrera <alvherre@commandprompt.com> The PostgreSQL Company - Command Prompt, Inc. PostgreSQL Replication, Consulting, Custom Development, 24x7 support