Thread: Shared memory usage calculations
I've had a look at the documentation for how much shared memory (in bytes) Postgres uses: http://www.postgresql.org/docs/8.4/static/kernel-resources.html#SHARED-MEMORY-PARAMETERS
However, after using these calculations to work out the shared memory usage for my own setup, the numbers I came up with are clearly wrong. Here's what I put my values down as:
max_locks_per_transaction = 64
max_connections = 220
autovacuum_max_workers = 3
max_prepared_transactions = 0
block_size = 8192
shared_buffers = 196000000 (196MB)
wal_block_size = 8192
wal_buffers = 8000000 (8MB)
Using the calculations on that page, it puts my shared memory requirements up to 1.7 terabytes, but my server is running along fine with these settings.
If I convert the sizes to kilobytes instead of bytes, it shows a total value of 47 megabytes, which, while not extreme, looks too low. And I am surprised that max_connections has relatively little bearing on the shared memory requirements. Is this right, or should is it more a case of it affecting semaphores? I was under the impression that the maximum number of connections played a large role in deciding shared memory limits.
My SHMMAX is currently set at 268435456, which I thought was just enough to cover my config settings.
Is the documentation flawed, or am I using the wrong units?
Thanks
Thom
Thom Brown <thombrown@gmail.com> writes: > I've had a look at the documentation for how much shared memory (in bytes) > Postgres uses: > http://www.postgresql.org/docs/8.4/static/kernel-resources.html#SHARED-MEMORY-PARAMETERS > However, after using these calculations to work out the shared memory usage > for my own setup, the numbers I came up with are clearly wrong. Clearly wrong compared to what --- ie, what's the actual size of your shared memory segment? (See ipcs) > If I convert the sizes to kilobytes instead of bytes, it shows a total value > of 47 megabytes, which, while not extreme, looks too low. And I am > surprised that max_connections has relatively little bearing on the shared > memory requirements. Is this right, or should is it more a case of it > affecting semaphores? I was under the impression that the maximum number of > connections played a large role in deciding shared memory limits. No, it doesn't really ... shared_buffers is the first-order component. Also, I'm not sure whether you realize that the native unit for these numbers is mostly *not* bytes. > shared_buffers = 196000000 (196MB) > wal_buffers = 8000000 (8MB) Neither of those parenthetical remarks are correct if that's exactly what you wrote in postgresql.conf. It might be worth checking the way these values are displayed in pg_settings. regards, tom lane
2009/10/10 Tom Lane <tgl@sss.pgh.pa.us>
Thom Brown <thombrown@gmail.com> writes:Clearly wrong compared to what --- ie, what's the actual size of your
> I've had a look at the documentation for how much shared memory (in bytes)
> Postgres uses:
> http://www.postgresql.org/docs/8.4/static/kernel-resources.html#SHARED-MEMORY-PARAMETERS
> However, after using these calculations to work out the shared memory usage
> for my own setup, the numbers I came up with are clearly wrong.
shared memory segment? (See ipcs)
I don't actually know the size of my shared memory segment. "sysctl -a | grep -i shmseg" doesn't return anything as it hasn't been set, and not sure how to find out the default value. I didn't see that mentioned in the calculations so I didn't take it into account anywhere. But 1.7TB shared memory can't be right as the entire system memory is only 720MB.
No, it doesn't really ... shared_buffers is the first-order component.
> If I convert the sizes to kilobytes instead of bytes, it shows a total value
> of 47 megabytes, which, while not extreme, looks too low. And I am
> surprised that max_connections has relatively little bearing on the shared
> memory requirements. Is this right, or should is it more a case of it
> affecting semaphores? I was under the impression that the maximum number of
> connections played a large role in deciding shared memory limits.
Also, I'm not sure whether you realize that the native unit for these
numbers is mostly *not* bytes.
> shared_buffers = 196000000 (196MB)
> wal_buffers = 8000000 (8MB)
Neither of those parenthetical remarks are correct if that's exactly
what you wrote in postgresql.conf. It might be worth checking the
way these values are displayed in pg_settings.
The parenthesised values are what I have in postgresql.conf. I take it these are supposed to be mebibytes then? When I check the value in pg_settings, it's set at 25088 with the unit set at 8kB is 196 mebibytes. It also shows wal_buffers at 1024 with 8kB units, which would be 8 mebibytes.
I gather I'm calculating the final value wrong though. :/
Thom
Thom Brown <thombrown@gmail.com> writes:
> 2009/10/10 Tom Lane <tgl@sss.pgh.pa.us>
>> Clearly wrong compared to what --- ie, what's the actual size of your
>> shared memory segment? (See ipcs)
> I don't actually know the size of my shared memory segment. "sysctl -a |
> grep -i shmseg" doesn't return anything as it hasn't been set, and not sure
> how to find out the default value.
There is no "default value", we are talking about the actual size of an
actual memory object. Which you find out with ipcs, not sysctl.
For instance
$ ipcs -a
------ Shared Memory Segments --------
key shmid owner perms bytes nattch status
0x00530201 3080195 tgl 600 37904384 4
------ Semaphore Arrays --------
key semid owner perms nsems
0x00530201 20873216 tgl 600 17
0x00530202 20905985 tgl 600 17
0x00530203 20938754 tgl 600 17
0x00530204 20971523 tgl 600 17
0x00530205 21004292 tgl 600 17
0x00530206 21037061 tgl 600 17
0x00530207 21069830 tgl 600 17
------ Message Queues --------
key msqid owner perms used-bytes messages
(I'm running this postmaster as "tgl", not the more usual "postgres".
On some platforms you might need to be the postgres user or root to see
shared objects belonging to postgres.)
regards, tom lane
2009/10/10 Tom Lane <tgl@sss.pgh.pa.us>
There is no "default value", we are talking about the actual size of an
actual memory object. Which you find out with ipcs, not sysctl.
For instance
$ ipcs -a
------ Shared Memory Segments --------
key shmid owner perms bytes nattch status
0x00530201 3080195 tgl 600 37904384 4
------ Semaphore Arrays --------
key semid owner perms nsems
0x00530201 20873216 tgl 600 17
0x00530202 20905985 tgl 600 17
0x00530203 20938754 tgl 600 17
0x00530204 20971523 tgl 600 17
0x00530205 21004292 tgl 600 17
0x00530206 21037061 tgl 600 17
0x00530207 21069830 tgl 600 17
------ Message Queues --------
key msqid owner perms used-bytes messages
(I'm running this postmaster as "tgl", not the more usual "postgres".
On some platforms you might need to be the postgres user or root to see
shared objects belonging to postgres.)
regards, tom lane
My mistake. I've never seen that before. Running it on mine returns the following:
------ Shared Memory Segments --------
key shmid owner perms bytes nattch status
0x0052e2c1 491520 postgres 600 235511808 4
------ Semaphore Arrays --------
key semid owner perms nsems
0x0052e2c1 8585218 postgres 600 17
0x0052e2c2 8617987 postgres 600 17
0x0052e2c3 8650756 postgres 600 17
0x0052e2c4 8683525 postgres 600 17
0x0052e2c5 8716294 postgres 600 17
0x0052e2c6 8749063 postgres 600 17
0x0052e2c7 8781832 postgres 600 17
0x0052e2c8 8814601 postgres 600 17
0x0052e2c9 8847370 postgres 600 17
0x0052e2ca 8880139 postgres 600 17
0x0052e2cb 8912908 postgres 600 17
0x0052e2cc 8945677 postgres 600 17
0x0052e2cd 8978446 postgres 600 17
0x0052e2ce 9011215 postgres 600 17
0x0052e2cf 9043984 postgres 600 17
0x0052e2d0 9076753 postgres 600 17
0x0052e2d1 9109522 postgres 600 17
0x0052e2d2 9142291 postgres 600 17
0x0052e2d3 9175060 postgres 600 17
0x0052e2d4 9207829 postgres 600 17
0x0052e2d5 9240598 postgres 600 17
0x0052e2d6 9273367 postgres 600 17
0x0052e2d7 9306136 postgres 600 17
0x0052e2d8 9338905 postgres 600 17
0x0052e2d9 9371674 postgres 600 17
0x0052e2da 9404443 postgres 600 17
0x0052e2db 9437212 postgres 600 17
0x0052e2dc 9469981 postgres 600 17
0x0052e2dd 9502750 postgres 600 17
0x0052e2de 9535519 postgres 600 17
0x0052e2df 9568288 postgres 600 17
0x0052e2e0 9601057 postgres 600 17
0x0052e2e1 9633826 postgres 600 17
0x0052e2e2 9666595 postgres 600 17
0x0052e2e3 9699364 postgres 600 17
0x0052e2e4 9732133 postgres 600 17
0x0052e2e5 9764902 postgres 600 17
0x0052e2e6 9797671 postgres 600 17
0x0052e2e7 9830440 postgres 600 17
0x0052e2e8 9863209 postgres 600 17
0x0052e2e9 9895978 postgres 600 17
0x0052e2ea 9928747 postgres 600 17
0x0052e2eb 9961516 postgres 600 17
0x0052e2ec 9994285 postgres 600 17
0x0052e2ed 10027054 postgres 600 17
0x0052e2ee 10059823 postgres 600 17
0x0052e2ef 10092592 postgres 600 17
0x0052e2f0 10125361 postgres 600 17
0x0052e2f1 10158130 postgres 600 17
0x0052e2f2 10190899 postgres 600 17
0x0052e2f3 10223668 postgres 600 17
0x0052e2f4 10256437 postgres 600 17
0x0052e2f5 10289206 postgres 600 17
0x0052e2f6 10321975 postgres 600 17
0x0052e2f7 10354744 postgres 600 17
0x0052e2f8 10387513 postgres 600 17
0x0052e2f9 10420282 postgres 600 17
0x0052e2fa 10453051 postgres 600 17
0x0052e2fb 10485820 postgres 600 17
0x0052e2fc 10518589 postgres 600 17
------ Message Queues --------
key msqid owner perms used-bytes messages
In any case, I gather that the documentation isn't inaccurate, rather my interpretation is. And having put my number of connections right up to 1500, Postgres still starts fine, so I'm not sure why I thought it would cause the shared memory limit to be reached so easily.
Thanks
Thom
Thom Brown <thombrown@gmail.com> writes:
> My mistake. I've never seen that before. Running it on mine returns the
> following:
> ------ Shared Memory Segments --------
> key shmid owner perms bytes nattch status
> 0x0052e2c1 491520 postgres 600 235511808 4
Okay, so your actual shmem size is 235MB, of which 196 is shared_buffers
and 8 is wal_buffers, and the rest is related to max_connections and
various third-order parameters. I'm not sure how precise the values
given in the docs are, but last I checked they weren't hugely wrong.
regards, tom lane