Thread: Memory issue on FreeBSD
Hi, I am running a PostgreSQL server on FreeBSD. The system has 32GB memory. Usually I use top to examine the memory usage of the system. After a while, a part, approximately 5GB, vanish from top, so that the memory rounds up to 27GB. After restarting PostgreSQL, I have all 32GB again available, but then it's already slightly decreasing. It's a standalone database server. It has an OpenStreetMap world database running with 353GB data (with indices). Some system information: # uname -r 9.0-RELEASE-p3 # pg_ctl --version pg_ctl (PostgreSQL) 9.1.6 # cat /boot/loader.conf ... kern.ipc.semmni=256 kern.ipc.semmns=512 kern.ipc.semmnu=256 kern.ipc.semumr=200 vm.pmap.shpgperproc=400 vm.pmap.pv_entry_max=50331648 ... # cat /pgdata/data/postgresql.conf ... default_statistics_target = 50 # pgtune wizard 2012-04-04 maintenance_work_mem = 4GB # pgtune wizard 2012-04-04 constraint_exclusion = on # pgtune wizard 2012-04-04 checkpoint_completion_target = 0.9 # pgtune wizard 2012-04-04 effective_cache_size = 24GB # pgtune wizard 2012-04-04 work_mem = 768MB # pgtune wizard 2012-04-04 wal_buffers = 16MB # pgtune wizard 2012-04-04 checkpoint_segments = 60 # 20 shared_buffers = 8GB # pgtune wizard 2012-04-04 max_connections = 100 synchronous_commit = off So any help finding out why my system "looses" some RAM is greatly appreciated :-) If more information is needed I will gladly provide it. Frank -- Frank BRONIEWSKI METRICO s.à r.l. géomètres technologies d'information géographique rue des Romains 36 L-5433 NIEDERDONVEN tél.: +352 26 74 94 - 28 fax.: +352 26 74 94 99 http://www.metrico.lu
ipcs in FreeBSD is a little ... tricky. ipcs -M ipcs -m ipcs -am could be your friends On Δευ 05 Νοε 2012 11:22:46 Frank Broniewski wrote: > Hi, > > I am running a PostgreSQL server on FreeBSD. The system has 32GB memory. > Usually I use top to examine the memory usage of the system. After a > while, a part, approximately 5GB, vanish from top, so that the memory > rounds up to 27GB. After restarting PostgreSQL, I have all 32GB again > available, but then it's already slightly decreasing. It's a standalone > database server. It has an OpenStreetMap world database running with > 353GB data (with indices). > > Some system information: > # uname -r > 9.0-RELEASE-p3 > # pg_ctl --version > pg_ctl (PostgreSQL) 9.1.6 > > # cat /boot/loader.conf > ... > kern.ipc.semmni=256 > kern.ipc.semmns=512 > kern.ipc.semmnu=256 > kern.ipc.semumr=200 > vm.pmap.shpgperproc=400 > vm.pmap.pv_entry_max=50331648 > ... > > # cat /pgdata/data/postgresql.conf > ... > default_statistics_target = 50 # pgtune wizard 2012-04-04 > maintenance_work_mem = 4GB # pgtune wizard 2012-04-04 > constraint_exclusion = on # pgtune wizard 2012-04-04 > checkpoint_completion_target = 0.9 # pgtune wizard 2012-04-04 > effective_cache_size = 24GB # pgtune wizard 2012-04-04 > work_mem = 768MB # pgtune wizard 2012-04-04 > wal_buffers = 16MB # pgtune wizard 2012-04-04 > checkpoint_segments = 60 # 20 > shared_buffers = 8GB # pgtune wizard 2012-04-04 > max_connections = 100 > synchronous_commit = off > > > So any help finding out why my system "looses" some RAM is greatly > appreciated :-) If more information is needed I will gladly provide it. > > Frank > > > > - Achilleas Mantzios IT DEPT
(scrap my previous internal email (hence fake) address this one is correct : sorry for that) You can stop pgsql, start it and then watch out for the increase in SEGSZ values. I pretty much think they are in bytes. I am pretty confident that this value depicts the shared_buffers size in bytes. On Δευ 05 Νοε 2012 13:14:37 Achilleas Mantzios wrote: > > ipcs in FreeBSD is a little ... tricky. > > ipcs -M > ipcs -m > ipcs -am > > could be your friends > > On Δευ 05 Νοε 2012 11:22:46 Frank Broniewski wrote: > > Hi, > > > > I am running a PostgreSQL server on FreeBSD. The system has 32GB memory. > > Usually I use top to examine the memory usage of the system. After a > > while, a part, approximately 5GB, vanish from top, so that the memory > > rounds up to 27GB. After restarting PostgreSQL, I have all 32GB again > > available, but then it's already slightly decreasing. It's a standalone > > database server. It has an OpenStreetMap world database running with > > 353GB data (with indices). > > > > Some system information: > > # uname -r > > 9.0-RELEASE-p3 > > # pg_ctl --version > > pg_ctl (PostgreSQL) 9.1.6 > > > > # cat /boot/loader.conf > > ... > > kern.ipc.semmni=256 > > kern.ipc.semmns=512 > > kern.ipc.semmnu=256 > > kern.ipc.semumr=200 > > vm.pmap.shpgperproc=400 > > vm.pmap.pv_entry_max=50331648 > > ... > > > > # cat /pgdata/data/postgresql.conf > > ... > > default_statistics_target = 50 # pgtune wizard 2012-04-04 > > maintenance_work_mem = 4GB # pgtune wizard 2012-04-04 > > constraint_exclusion = on # pgtune wizard 2012-04-04 > > checkpoint_completion_target = 0.9 # pgtune wizard 2012-04-04 > > effective_cache_size = 24GB # pgtune wizard 2012-04-04 > > work_mem = 768MB # pgtune wizard 2012-04-04 > > wal_buffers = 16MB # pgtune wizard 2012-04-04 > > checkpoint_segments = 60 # 20 > > shared_buffers = 8GB # pgtune wizard 2012-04-04 > > max_connections = 100 > > synchronous_commit = off > > > > > > So any help finding out why my system "looses" some RAM is greatly > > appreciated :-) If more information is needed I will gladly provide it. > > > > Frank > > > > > > > > > - > Achilleas Mantzios > IT DEPT > > > - Achilleas Mantzios IT DEPT
Hi,
thank you for your feedback. I had a look at those commands and their
output, especially in conjunction with the SEGSZ value from icps -am
Here's an example output:
# ipcs -am
Shared Memory:
T ID KEY MODE OWNER GROUP CREATOR
CGROUP NATTCH SEGSZ CPID LPID ATIME
DTIME CTIME
m 262144 5432001 --rw------- pgsql pgsql pgsql pgsql
12 8813993984 45512 45512 13:49:28
14:31:29 13:49:28
but frankly this tells me nothing. I can tell that the value SEGSZ is
right from the start 8813993984 and it doesn't change anymore. The only
value that changes is the NATTCH value, I observed a range from 8 to 36
there. I agree that the SEGSZ value matches the 8GB shared buffer, but
how can I make the connection of my 5GB missing in top? I wonder if this
might be the maintenance_work_mem, which is set to 4GB?
Many thanks,
Frank
Am 2012-11-05 12:14, schrieb Achilleas Mantzios:
>
> ipcs in FreeBSD is a little ... tricky.
>
> ipcs -M
> ipcs -m
> ipcs -am
>
> could be your friends
>
> On Δευ 05 Νοε 2012 11:22:46 Frank Broniewski wrote:
>> Hi,
>>
>> I am running a PostgreSQL server on FreeBSD. The system has 32GB memory.
>> Usually I use top to examine the memory usage of the system. After a
>> while, a part, approximately 5GB, vanish from top, so that the memory
>> rounds up to 27GB. After restarting PostgreSQL, I have all 32GB again
>> available, but then it's already slightly decreasing. It's a standalone
>> database server. It has an OpenStreetMap world database running with
>> 353GB data (with indices).
>>
>> Some system information:
>> # uname -r
>> 9.0-RELEASE-p3
>> # pg_ctl --version
>> pg_ctl (PostgreSQL) 9.1.6
>>
>> # cat /boot/loader.conf
>> ...
>> kern.ipc.semmni=256
>> kern.ipc.semmns=512
>> kern.ipc.semmnu=256
>> kern.ipc.semumr=200
>> vm.pmap.shpgperproc=400
>> vm.pmap.pv_entry_max=50331648
>> ...
>>
>> # cat /pgdata/data/postgresql.conf
>> ...
>> default_statistics_target = 50 # pgtune wizard 2012-04-04
>> maintenance_work_mem = 4GB # pgtune wizard 2012-04-04
>> constraint_exclusion = on # pgtune wizard 2012-04-04
>> checkpoint_completion_target = 0.9 # pgtune wizard 2012-04-04
>> effective_cache_size = 24GB # pgtune wizard 2012-04-04
>> work_mem = 768MB # pgtune wizard 2012-04-04
>> wal_buffers = 16MB # pgtune wizard 2012-04-04
>> checkpoint_segments = 60 # 20
>> shared_buffers = 8GB # pgtune wizard 2012-04-04
>> max_connections = 100
>> synchronous_commit = off
>>
>>
>> So any help finding out why my system "looses" some RAM is greatly
>> appreciated :-) If more information is needed I will gladly provide it.
>>
>> Frank
>>
>>
>>
>>
> -
> Achilleas Mantzios
> IT DEPT
>
>
--
Frank BRONIEWSKI
METRICO s.à r.l.
géomètres
technologies d'information géographique
rue des Romains 36
L-5433 NIEDERDONVEN
tél.: +352 26 74 94 - 28
fax.: +352 26 74 94 99
http://www.metrico.lu
How do you measure that smth is missing from top? What values do you add? I am currently running 8.3 but we shouldn't be so far apart top-wise. What is the reading under SIZE and RES in top for all postgresql processes? Take note that shared mem should be recorded for each and every postmaster running. On Δευ 05 Νοε 2012 14:36:44 Frank Broniewski wrote: > Hi, > > thank you for your feedback. I had a look at those commands and their > output, especially in conjunction with the SEGSZ value from icps -am > > Here's an example output: > # ipcs -am > Shared Memory: > T ID KEY MODE OWNER GROUP CREATOR > CGROUP NATTCH SEGSZ CPID LPID ATIME > DTIME CTIME > m 262144 5432001 --rw------- pgsql pgsql pgsql pgsql > 12 8813993984 45512 45512 13:49:28 > 14:31:29 13:49:28 > > but frankly this tells me nothing. I can tell that the value SEGSZ is > right from the start 8813993984 and it doesn't change anymore. The only > value that changes is the NATTCH value, I observed a range from 8 to 36 > there. I agree that the SEGSZ value matches the 8GB shared buffer, but > how can I make the connection of my 5GB missing in top? I wonder if this > might be the maintenance_work_mem, which is set to 4GB? > > Many thanks, > > Frank > > Am 2012-11-05 12:14, schrieb Achilleas Mantzios: > > > > ipcs in FreeBSD is a little ... tricky. > > > > ipcs -M > > ipcs -m > > ipcs -am > > > > could be your friends > > > > On Δευ 05 Νοε 2012 11:22:46 Frank Broniewski wrote: > >> Hi, > >> > >> I am running a PostgreSQL server on FreeBSD. The system has 32GB memory. > >> Usually I use top to examine the memory usage of the system. After a > >> while, a part, approximately 5GB, vanish from top, so that the memory > >> rounds up to 27GB. After restarting PostgreSQL, I have all 32GB again > >> available, but then it's already slightly decreasing. It's a standalone > >> database server. It has an OpenStreetMap world database running with > >> 353GB data (with indices). > >> > >> Some system information: > >> # uname -r > >> 9.0-RELEASE-p3 > >> # pg_ctl --version > >> pg_ctl (PostgreSQL) 9.1.6 > >> > >> # cat /boot/loader.conf > >> ... > >> kern.ipc.semmni=256 > >> kern.ipc.semmns=512 > >> kern.ipc.semmnu=256 > >> kern.ipc.semumr=200 > >> vm.pmap.shpgperproc=400 > >> vm.pmap.pv_entry_max=50331648 > >> ... > >> > >> # cat /pgdata/data/postgresql.conf > >> ... > >> default_statistics_target = 50 # pgtune wizard 2012-04-04 > >> maintenance_work_mem = 4GB # pgtune wizard 2012-04-04 > >> constraint_exclusion = on # pgtune wizard 2012-04-04 > >> checkpoint_completion_target = 0.9 # pgtune wizard 2012-04-04 > >> effective_cache_size = 24GB # pgtune wizard 2012-04-04 > >> work_mem = 768MB # pgtune wizard 2012-04-04 > >> wal_buffers = 16MB # pgtune wizard 2012-04-04 > >> checkpoint_segments = 60 # 20 > >> shared_buffers = 8GB # pgtune wizard 2012-04-04 > >> max_connections = 100 > >> synchronous_commit = off > >> > >> > >> So any help finding out why my system "looses" some RAM is greatly > >> appreciated :-) If more information is needed I will gladly provide it. > >> > >> Frank > >> > >> > >> > >> > > - > > Achilleas Mantzios > > IT DEPT > > > > > > > - Achilleas Mantzios IT DEPT
Hi, I just add the different memory values together (minus the buffers). Usually this sums up (+/-) to the installed memory size, at least on my other machines. I found a thread similar to my problem here [1], but no solution. I don't mind top showing false values, but if there's a larger problem behind this, then I really want to solve it. Top is really just an indicator for this issue, it's also visible in my munin stats [2] Below is a output _without_ postgresql running: Mem: 59M Active, 17G Inact, 3953M Wired, 1325M Cache, 3283M Buf, 8663M Free Swap: 4096M Total, 828K Used, 4095M Free and this is after a few hours of running: Mem: 91M Active, 17G Inact, 3983M Wired, 1526M Cache, 3283M Buf, 155M Free Swap: 4096M Total, 828K Used, 4095M Free some memory related sysctl values: hw.realmem: 34879832064 hw.physmem: 34322804736 hw.usermem: 30161108992 # sysctl vm.vmtotal vm.vmtotal: System wide totals computed every five seconds: (values in kilobytes) =============================================== Processes: (RUNQ: 1 Disk Wait: 0 Page Wait: 0 Sleep: 70) Virtual Memory: (Total: 1084659688K Active: 10400940K) Real Memory: (Total: 1616176K Active: 1349052K) Shared Virtual Memory: (Total: 60840K Active: 14132K) Shared Real Memory: (Total: 11644K Active: 8388K) Free Memory Pages: 7263972K [1] http://lists.freebsd.org/pipermail/freebsd-stable/2011-January/061247.html [2] http://www.gis-hosting.lu/monitor/munin/metrico/bilbo.metrico/memory.html Am 2012-11-05 15:21, schrieb Achilleas Mantzios: > How do you measure that smth is missing from top? What values do you add? > I am currently running 8.3 but we shouldn't be so far apart top-wise. > What is the reading under SIZE and RES in top for all postgresql processes? > Take note that shared mem should be recorded for each and every postmaster running. > > On Δευ 05 Νοε 2012 14:36:44 Frank Broniewski wrote: >> Hi, >> >> thank you for your feedback. I had a look at those commands and their >> output, especially in conjunction with the SEGSZ value from icps -am >> >> Here's an example output: >> # ipcs -am >> Shared Memory: >> T ID KEY MODE OWNER GROUP CREATOR >> CGROUP NATTCH SEGSZ CPID LPID ATIME >> DTIME CTIME >> m 262144 5432001 --rw------- pgsql pgsql pgsql pgsql >> 12 8813993984 45512 45512 13:49:28 >> 14:31:29 13:49:28 >> >> but frankly this tells me nothing. I can tell that the value SEGSZ is >> right from the start 8813993984 and it doesn't change anymore. The only >> value that changes is the NATTCH value, I observed a range from 8 to 36 >> there. I agree that the SEGSZ value matches the 8GB shared buffer, but >> how can I make the connection of my 5GB missing in top? I wonder if this >> might be the maintenance_work_mem, which is set to 4GB? >> >> Many thanks, >> >> Frank >> >> Am 2012-11-05 12:14, schrieb Achilleas Mantzios: >>> >>> ipcs in FreeBSD is a little ... tricky. >>> >>> ipcs -M >>> ipcs -m >>> ipcs -am >>> >>> could be your friends >>> >>> On Δευ 05 Νοε 2012 11:22:46 Frank Broniewski wrote: >>>> Hi, >>>> >>>> I am running a PostgreSQL server on FreeBSD. The system has 32GB memory. >>>> Usually I use top to examine the memory usage of the system. After a >>>> while, a part, approximately 5GB, vanish from top, so that the memory >>>> rounds up to 27GB. After restarting PostgreSQL, I have all 32GB again >>>> available, but then it's already slightly decreasing. It's a standalone >>>> database server. It has an OpenStreetMap world database running with >>>> 353GB data (with indices). >>>> >>>> Some system information: >>>> # uname -r >>>> 9.0-RELEASE-p3 >>>> # pg_ctl --version >>>> pg_ctl (PostgreSQL) 9.1.6 >>>> >>>> # cat /boot/loader.conf >>>> ... >>>> kern.ipc.semmni=256 >>>> kern.ipc.semmns=512 >>>> kern.ipc.semmnu=256 >>>> kern.ipc.semumr=200 >>>> vm.pmap.shpgperproc=400 >>>> vm.pmap.pv_entry_max=50331648 >>>> ... >>>> >>>> # cat /pgdata/data/postgresql.conf >>>> ... >>>> default_statistics_target = 50 # pgtune wizard 2012-04-04 >>>> maintenance_work_mem = 4GB # pgtune wizard 2012-04-04 >>>> constraint_exclusion = on # pgtune wizard 2012-04-04 >>>> checkpoint_completion_target = 0.9 # pgtune wizard 2012-04-04 >>>> effective_cache_size = 24GB # pgtune wizard 2012-04-04 >>>> work_mem = 768MB # pgtune wizard 2012-04-04 >>>> wal_buffers = 16MB # pgtune wizard 2012-04-04 >>>> checkpoint_segments = 60 # 20 >>>> shared_buffers = 8GB # pgtune wizard 2012-04-04 >>>> max_connections = 100 >>>> synchronous_commit = off >>>> >>>> >>>> So any help finding out why my system "looses" some RAM is greatly >>>> appreciated :-) If more information is needed I will gladly provide it. >>>> >>>> Frank >>>> >>>> >>>> >>>> >>> - >>> Achilleas Mantzios >>> IT DEPT >>> >>> >> >> >> > - > Achilleas Mantzios > IT DEPT > > -- Frank BRONIEWSKI METRICO s.à r.l. géomètres technologies d'information géographique rue des Romains 36 L-5433 NIEDERDONVEN tél.: +352 26 74 94 - 28 fax.: +352 26 74 94 99 http://www.metrico.lu
Since the top reporting goes back to normal when postgresql is stopped , and since postgresql is special due to the use of IPC, i would be inclined to think that the culprit here is the shared memory. I don't know where maintenance_work_mem really lives (process normal address space or IPC shared mem) and if that makes any difference. If it is possible you might play with those two values and see if anything changes. Currently i have : maintenance_work_mem = 960MB # pgtune wizard 2012-11-01 shared_buffers = 3840MB # pgtune wizard 2012-11-01 top: last pid: 74896; load averages: 0.02, 0.08, 0.08 up 4+06:20:31 18:14:19 187 processes: 1 running, 172 sleeping, 14 zombie CPU: % user, % nice, % system, % interrupt, % idle Mem: 4064M Active, 8111M Inact, 2014M Wired, 322M Cache, 1645M Buf, 1106M Free Swap: 8000M Total, 608K Used, 7999M Free hw.physmem: 17144205312 hw.usermem: 15028662272 hw.realmem: 17985175552 top (excluding Buf) amounts to 15617 Megs while physmem shows as 16349 Megs but as i said i run 8.3 on AMD64 and pgsql 9.2.1 On Δευ 05 Νοε 2012 16:11:39 Frank Broniewski wrote: > Hi, > > I just add the different memory values together (minus the buffers). > Usually this sums up (+/-) to the installed memory size, at least on my > other machines. I found a thread similar to my problem here [1], but no > solution. I don't mind top showing false values, but if there's a larger > problem behind this, then I really want to solve it. > > Top is really just an indicator for this issue, it's also visible in my > munin stats [2] > > Below is a output _without_ postgresql running: > Mem: 59M Active, 17G Inact, 3953M Wired, 1325M Cache, 3283M Buf, 8663M Free > Swap: 4096M Total, 828K Used, 4095M Free > > > and this is after a few hours of running: > > Mem: 91M Active, 17G Inact, 3983M Wired, 1526M Cache, 3283M Buf, 155M Free > Swap: 4096M Total, 828K Used, 4095M Free > > some memory related sysctl values: > hw.realmem: 34879832064 > hw.physmem: 34322804736 > hw.usermem: 30161108992 > > # sysctl vm.vmtotal > vm.vmtotal: > System wide totals computed every five seconds: (values in kilobytes) > =============================================== > Processes: (RUNQ: 1 Disk Wait: 0 Page Wait: 0 Sleep: 70) > Virtual Memory: (Total: 1084659688K Active: 10400940K) > Real Memory: (Total: 1616176K Active: 1349052K) > Shared Virtual Memory: (Total: 60840K Active: 14132K) > Shared Real Memory: (Total: 11644K Active: 8388K) > Free Memory Pages: 7263972K > > > [1] > http://lists.freebsd.org/pipermail/freebsd-stable/2011-January/061247.html > [2] > http://www.gis-hosting.lu/monitor/munin/metrico/bilbo.metrico/memory.html > > > Am 2012-11-05 15:21, schrieb Achilleas Mantzios: > > How do you measure that smth is missing from top? What values do you add? > > I am currently running 8.3 but we shouldn't be so far apart top-wise. > > What is the reading under SIZE and RES in top for all postgresql processes? > > Take note that shared mem should be recorded for each and every postmaster running. > > > > On Δευ 05 Νοε 2012 14:36:44 Frank Broniewski wrote: > >> Hi, > >> > >> thank you for your feedback. I had a look at those commands and their > >> output, especially in conjunction with the SEGSZ value from icps -am > >> > >> Here's an example output: > >> # ipcs -am > >> Shared Memory: > >> T ID KEY MODE OWNER GROUP CREATOR > >> CGROUP NATTCH SEGSZ CPID LPID ATIME > >> DTIME CTIME > >> m 262144 5432001 --rw------- pgsql pgsql pgsql pgsql > >> 12 8813993984 45512 45512 13:49:28 > >> 14:31:29 13:49:28 > >> > >> but frankly this tells me nothing. I can tell that the value SEGSZ is > >> right from the start 8813993984 and it doesn't change anymore. The only > >> value that changes is the NATTCH value, I observed a range from 8 to 36 > >> there. I agree that the SEGSZ value matches the 8GB shared buffer, but > >> how can I make the connection of my 5GB missing in top? I wonder if this > >> might be the maintenance_work_mem, which is set to 4GB? > >> > >> Many thanks, > >> > >> Frank > >> > >> Am 2012-11-05 12:14, schrieb Achilleas Mantzios: > >>> > >>> ipcs in FreeBSD is a little ... tricky. > >>> > >>> ipcs -M > >>> ipcs -m > >>> ipcs -am > >>> > >>> could be your friends > >>> > >>> On Δευ 05 Νοε 2012 11:22:46 Frank Broniewski wrote: > >>>> Hi, > >>>> > >>>> I am running a PostgreSQL server on FreeBSD. The system has 32GB memory. > >>>> Usually I use top to examine the memory usage of the system. After a > >>>> while, a part, approximately 5GB, vanish from top, so that the memory > >>>> rounds up to 27GB. After restarting PostgreSQL, I have all 32GB again > >>>> available, but then it's already slightly decreasing. It's a standalone > >>>> database server. It has an OpenStreetMap world database running with > >>>> 353GB data (with indices). > >>>> > >>>> Some system information: > >>>> # uname -r > >>>> 9.0-RELEASE-p3 > >>>> # pg_ctl --version > >>>> pg_ctl (PostgreSQL) 9.1.6 > >>>> > >>>> # cat /boot/loader.conf > >>>> ... > >>>> kern.ipc.semmni=256 > >>>> kern.ipc.semmns=512 > >>>> kern.ipc.semmnu=256 > >>>> kern.ipc.semumr=200 > >>>> vm.pmap.shpgperproc=400 > >>>> vm.pmap.pv_entry_max=50331648 > >>>> ... > >>>> > >>>> # cat /pgdata/data/postgresql.conf > >>>> ... > >>>> default_statistics_target = 50 # pgtune wizard 2012-04-04 > >>>> maintenance_work_mem = 4GB # pgtune wizard 2012-04-04 > >>>> constraint_exclusion = on # pgtune wizard 2012-04-04 > >>>> checkpoint_completion_target = 0.9 # pgtune wizard 2012-04-04 > >>>> effective_cache_size = 24GB # pgtune wizard 2012-04-04 > >>>> work_mem = 768MB # pgtune wizard 2012-04-04 > >>>> wal_buffers = 16MB # pgtune wizard 2012-04-04 > >>>> checkpoint_segments = 60 # 20 > >>>> shared_buffers = 8GB # pgtune wizard 2012-04-04 > >>>> max_connections = 100 > >>>> synchronous_commit = off > >>>> > >>>> > >>>> So any help finding out why my system "looses" some RAM is greatly > >>>> appreciated :-) If more information is needed I will gladly provide it. > >>>> > >>>> Frank > >>>> > >>>> > >>>> > >>>> > >>> - > >>> Achilleas Mantzios > >>> IT DEPT > >>> > >>> > >> > >> > >> > > - > > Achilleas Mantzios > > IT DEPT > > > > > > > - Achilleas Mantzios IT DEPT
On Mon, Nov 5, 2012 at 10:11 AM, Frank Broniewski <brfr@metrico.lu> wrote:
and this is after a few hours of running:
Mem: 91M Active, 17G Inact, 3983M Wired, 1526M Cache, 3283M Buf, 155M Free
Swap: 4096M Total, 828K Used, 4095M Free
For comparison, here is the output of a 32GB FreeBSD 9.0/amd64 server, with Postgres 9.0.7 running since June 10, and is heavily pounded on 24x7. The data + indexes are about 240GB on disk. This server only runs postgres aside from the basic system processes.
Mem: 231M Active, 21G Inact, 3777M Wired, 1009M Cache, 3285M Buf, 191M Free
Swap: 4096M Total, 272K Used, 4096M Free
I agree with the conclusion that the shared memory segments are confusing the output of top. There are no memory leaks, and FreeBSD doesn't "lose" any memory.
There are some scripts floating around that read values from sysctl vm.stats.vm and format them nicely to tell you how much memory is used up and free. Try the one referenced here: http://www.cyberciti.biz/faq/freebsd-command-to-get-ram-information/
Vick,
fantastic script, thanx! FreeBSD sysctl system is awesome!
On Τρι 06 Νοε 2012 14:33:43 Vick Khera wrote:
On Mon, Nov 5, 2012 at 10:11 AM, Frank Broniewski <brfr@metrico.lu> wrote:
and this is after a few hours of running:
Mem: 91M Active, 17G Inact, 3983M Wired, 1526M Cache, 3283M Buf, 155M Free
Swap: 4096M Total, 828K Used, 4095M Free
For comparison, here is the output of a 32GB FreeBSD 9.0/amd64 server, with Postgres 9.0.7 running since June 10, and is heavily pounded on 24x7. The data + indexes are about 240GB on disk. This server only runs postgres aside from the basic system processes.
Mem: 231M Active, 21G Inact, 3777M Wired, 1009M Cache, 3285M Buf, 191M Free
Swap: 4096M Total, 272K Used, 4096M Free
I agree with the conclusion that the shared memory segments are confusing the output of top. There are no memory leaks, and FreeBSD doesn't "lose" any memory.
There are some scripts floating around that read values from sysctl vm.stats.vm and format them nicely to tell you how much memory is used up and free. Try the one referenced here: http://www.cyberciti.biz/faq/freebsd-command-to-get-ram-information/
-
Achilleas Mantzios
IT DEPT
Hey, this is really cool. I directly tried the script and there's a line from the output that caught my eye: > mem_gap_vm: + 8812892160 ( 8404MB) [ 26%] Memory gap: UNKNOWN is this the shared buffers? I guess so, but I want to confirm my guess ... Frank Am 2012-11-07 09:26, schrieb Achilleas Mantzios: > Vick, > fantastic script, thanx! FreeBSD sysctl system is awesome! > > On Τρι 06 Νοε 2012 14:33:43 Vick Khera wrote: > > > > On Mon, Nov 5, 2012 at 10:11 AM, Frank Broniewski <brfr@metrico.lu> wrote: > > and this is after a few hours of running: > > Mem: 91M Active, 17G Inact, 3983M Wired, 1526M Cache, 3283M Buf, 155M Free > Swap: 4096M Total, 828K Used, 4095M Free > > > > > For comparison, here is the output of a 32GB FreeBSD 9.0/amd64 server, with Postgres 9.0.7 running since June 10, and isheavily pounded on 24x7. The data + indexes are about 240GB on disk. This server only runs postgres aside from the basicsystem processes. > > > Mem: 231M Active, 21G Inact, 3777M Wired, 1009M Cache, 3285M Buf, 191M Free > Swap: 4096M Total, 272K Used, 4096M Free > > > I agree with the conclusion that the shared memory segments are confusing the output of top. There are no memory leaks,and FreeBSD doesn't "lose" any memory. > > > There are some scripts floating around that read values from sysctl vm.stats.vm and format them nicely to tell you howmuch memory is used up and free. Try the one referenced here: http://www.cyberciti.biz/faq/freebsd-command-to-get-ram-information/ > > > > > > - > Achilleas Mantzios > IT DEPT > -- Frank BRONIEWSKI METRICO s.à r.l. géomètres technologies d'information géographique rue des Romains 36 L-5433 NIEDERDONVEN tél.: +352 26 74 94 - 28 fax.: +352 26 74 94 99 http://www.metrico.lu
On Τετ 07 Νοε 2012 09:42:47 Frank Broniewski wrote:
> Hey, this is really cool. I directly tried the script and there's a line
> from the output that caught my eye:
>
> > mem_gap_vm: + 8812892160 ( 8404MB) [ 26%] Memory gap: UNKNOWN
>
> is this the shared buffers? I guess so, but I want to confirm my guess ...
Hmm, that would be ideal, (from an understanding perspective) but at least in my system (FreeBSD-8.3), no.
psql -q -t -c "show shared_buffers" | grep -v -e '^$' | awk '{print $1}'
3840MB
SYSTEM MEMORY INFORMATION:
mem_gap_vm: + 996843520 ( 950MB) [ 5%] Memory gap: UNKNOWN
$mem_gap_vm = $mem_all - ($mem_wire + $mem_active + $mem_inactive + $mem_cache + $mem_free);
mem_all is some rounded and more rationalized version less than hw.physmem : $mem_all = $sysctl->{"vm.stats.vm.v_page_count"} * $sysctl->{"hw.pagesize"};
Anyway, this is not so postgresql related at the moment. The correct thing to do (since you run production servers on FreeBSD) is to post to the relevant
FreeBSD list and/or forum. freebsd-questions@freebsd.org and freebsd-stable@freebsd.org would be a good start.
Also the forums : http://forums.freebsd.org/forumdisplay.php?f=3
Only after gathering substantial info from there, would it make sense to come back here and maybe ask more questions.
And since we are observing different percentages of gaps (mine is 5%, yours is 26%), i think maybe you should look into it on the FreeBSD camp.
Please drop the link to the relevant thread there, if you decide to do so.
I would like to follow this.
Thanx!
>
> Frank
>
> Am 2012-11-07 09:26, schrieb Achilleas Mantzios:
> > Vick,
> > fantastic script, thanx! FreeBSD sysctl system is awesome!
> >
> > On Τρι 06 Νοε 2012 14:33:43 Vick Khera wrote:
> >
> >
> >
> > On Mon, Nov 5, 2012 at 10:11 AM, Frank Broniewski <brfr@metrico.lu> wrote:
> >
> > and this is after a few hours of running:
> >
> > Mem: 91M Active, 17G Inact, 3983M Wired, 1526M Cache, 3283M Buf, 155M Free
> > Swap: 4096M Total, 828K Used, 4095M Free
> >
> >
> >
> >
> > For comparison, here is the output of a 32GB FreeBSD 9.0/amd64 server, with Postgres 9.0.7 running since June 10, and is heavily pounded on 24x7. The data + indexes are about 240GB on disk. This server only runs postgres aside from the basic system processes.
> >
> >
> > Mem: 231M Active, 21G Inact, 3777M Wired, 1009M Cache, 3285M Buf, 191M Free
> > Swap: 4096M Total, 272K Used, 4096M Free
> >
> >
> > I agree with the conclusion that the shared memory segments are confusing the output of top. There are no memory leaks, and FreeBSD doesn't "lose" any memory.
> >
> >
> > There are some scripts floating around that read values from sysctl vm.stats.vm and format them nicely to tell you how much memory is used up and free. Try the one referenced here: http://www.cyberciti.biz/faq/freebsd-command-to-get-ram-information/
> >
> >
> >
> >
> >
> > -
> > Achilleas Mantzios
> > IT DEPT
> >
>
>
>
-
Achilleas Mantzios
IT DEPT
Vick,
fantastic script, thanx! FreeBSD sysctl system is awesome!
On Τρι 06 Νοε 2012 14:33:43 Vick Khera wrote:
On Mon, Nov 5, 2012 at 10:11 AM, Frank Broniewski <brfr@metrico.lu> wrote:
and this is after a few hours of running:
Mem: 91M Active, 17G Inact, 3983M Wired, 1526M Cache, 3283M Buf, 155M Free
Swap: 4096M Total, 828K Used, 4095M Free
For comparison, here is the output of a 32GB FreeBSD 9.0/amd64 server, with Postgres 9.0.7 running since June 10, and is heavily pounded on 24x7. The data + indexes are about 240GB on disk. This server only runs postgres aside from the basic system processes.
Mem: 231M Active, 21G Inact, 3777M Wired, 1009M Cache, 3285M Buf, 191M Free
Swap: 4096M Total, 272K Used, 4096M Free
I agree with the conclusion that the shared memory segments are confusing the output of top. There are no memory leaks, and FreeBSD doesn't "lose" any memory.
There are some scripts floating around that read values from sysctl vm.stats.vm and format them nicely to tell you how much memory is used up and free. Try the one referenced here: http://www.cyberciti.biz/faq/freebsd-command-to-get-ram-information/
-
Achilleas Mantzios
IT DEPT
FYI http://freebsd.1045724.n5.nabble.com/Postgresql-related-memory-question-td5759467.html Am 2012-11-07 10:28, schrieb Achilleas Mantzios: > On Τετ 07 Νοε 2012 09:42:47 Frank Broniewski wrote: >> Hey, this is really cool. I directly tried the script and there's a line >> from the output that caught my eye: >> >> > mem_gap_vm: + 8812892160 ( 8404MB) [ 26%] Memory gap: UNKNOWN >> >> is this the shared buffers? I guess so, but I want to confirm my guess ... > > Hmm, that would be ideal, (from an understanding perspective) but at least in my system (FreeBSD-8.3), no. > > psql -q -t -c "show shared_buffers" | grep -v -e '^$' | awk '{print $1}' > 3840MB > > SYSTEM MEMORY INFORMATION: > mem_gap_vm: + 996843520 ( 950MB) [ 5%] Memory gap: UNKNOWN > > $mem_gap_vm = $mem_all - ($mem_wire + $mem_active + $mem_inactive + $mem_cache + $mem_free); > > mem_all is some rounded and more rationalized version less than hw.physmem : $mem_all = $sysctl->{"vm.stats.vm.v_page_count"}* $sysctl->{"hw.pagesize"}; > > Anyway, this is not so postgresql related at the moment. The correct thing to do (since you run production servers on FreeBSD)is to post to the relevant > FreeBSD list and/or forum. freebsd-questions@freebsd.org and freebsd-stable@freebsd.org would be a good start. > Also the forums : http://forums.freebsd.org/forumdisplay.php?f=3 > Only after gathering substantial info from there, would it make sense to come back here and maybe ask more questions. > And since we are observing different percentages of gaps (mine is 5%, yours is 26%), i think maybe you should look intoit on the FreeBSD camp. > > Please drop the link to the relevant thread there, if you decide to do so. > > I would like to follow this. > > Thanx! > >> >> Frank >> >> Am 2012-11-07 09:26, schrieb Achilleas Mantzios: >>> Vick, >>> fantastic script, thanx! FreeBSD sysctl system is awesome! >>> >>> On Τρι 06 Νοε 2012 14:33:43 Vick Khera wrote: >>> >>> >>> >>> On Mon, Nov 5, 2012 at 10:11 AM, Frank Broniewski <brfr@metrico.lu> wrote: >>> >>> and this is after a few hours of running: >>> >>> Mem: 91M Active, 17G Inact, 3983M Wired, 1526M Cache, 3283M Buf, 155M Free >>> Swap: 4096M Total, 828K Used, 4095M Free >>> >>> >>> >>> >>> For comparison, here is the output of a 32GB FreeBSD 9.0/amd64 server, with Postgres 9.0.7 running since June 10, andis heavily pounded on 24x7. The data + indexes are about 240GB on disk. This server only runs postgres aside from thebasic system processes. >>> >>> >>> Mem: 231M Active, 21G Inact, 3777M Wired, 1009M Cache, 3285M Buf, 191M Free >>> Swap: 4096M Total, 272K Used, 4096M Free >>> >>> >>> I agree with the conclusion that the shared memory segments are confusing the output of top. There are no memory leaks,and FreeBSD doesn't "lose" any memory. >>> >>> >>> There are some scripts floating around that read values from sysctl vm.stats.vm and format them nicely to tell you howmuch memory is used up and free. Try the one referenced here: http://www.cyberciti.biz/faq/freebsd-command-to-get-ram-information/ >>> >>> >>> >>> >>> >>> - >>> Achilleas Mantzios >>> IT DEPT >>> >> >> >> > - > Achilleas Mantzios > IT DEPT > -- Frank BRONIEWSKI METRICO s.à r.l. géomètres technologies d'information géographique rue des Romains 36 L-5433 NIEDERDONVEN tél.: +352 26 74 94 - 28 fax.: +352 26 74 94 99 http://www.metrico.lu
Thanx for the link. I just think that it would be a good idea, instead of posting the links at this list, to include a dense but detailed summary of the situation in your machine, and give as much data as possible. In short, you might do a quantitative compilation of this thread, and present it in a nice way in order to gain more attention. Also, i think posting to -stable would be a better idea, -questions is for noobs. On Παρ 09 Νοε 2012 09:37:14 Frank Broniewski wrote: > FYI > http://freebsd.1045724.n5.nabble.com/Postgresql-related-memory-question-td5759467.html > > > Am 2012-11-07 10:28, schrieb Achilleas Mantzios: > > On Τετ 07 Νοε 2012 09:42:47 Frank Broniewski wrote: > >> Hey, this is really cool. I directly tried the script and there's a line > >> from the output that caught my eye: > >> > >> > mem_gap_vm: + 8812892160 ( 8404MB) [ 26%] Memory gap: UNKNOWN > >> > >> is this the shared buffers? I guess so, but I want to confirm my guess ... > > > > Hmm, that would be ideal, (from an understanding perspective) but at least in my system (FreeBSD-8.3), no. > > > > psql -q -t -c "show shared_buffers" | grep -v -e '^$' | awk '{print $1}' > > 3840MB > > > > SYSTEM MEMORY INFORMATION: > > mem_gap_vm: + 996843520 ( 950MB) [ 5%] Memory gap: UNKNOWN > > > > $mem_gap_vm = $mem_all - ($mem_wire + $mem_active + $mem_inactive + $mem_cache + $mem_free); > > > > mem_all is some rounded and more rationalized version less than hw.physmem : $mem_all = $sysctl->{"vm.stats.vm.v_page_count"}* $sysctl->{"hw.pagesize"}; > > > > Anyway, this is not so postgresql related at the moment. The correct thing to do (since you run production servers onFreeBSD) is to post to the relevant > > FreeBSD list and/or forum. freebsd-questions@freebsd.org and freebsd-stable@freebsd.org would be a good start. > > Also the forums : http://forums.freebsd.org/forumdisplay.php?f=3 > > Only after gathering substantial info from there, would it make sense to come back here and maybe ask more questions. > > And since we are observing different percentages of gaps (mine is 5%, yours is 26%), i think maybe you should look intoit on the FreeBSD camp. > > > > Please drop the link to the relevant thread there, if you decide to do so. > > > > I would like to follow this. > > > > Thanx! > > > >> > >> Frank > >> > >> Am 2012-11-07 09:26, schrieb Achilleas Mantzios: > >>> Vick, > >>> fantastic script, thanx! FreeBSD sysctl system is awesome! > >>> > >>> On Τρι 06 Νοε 2012 14:33:43 Vick Khera wrote: > >>> > >>> > >>> > >>> On Mon, Nov 5, 2012 at 10:11 AM, Frank Broniewski <brfr@metrico.lu> wrote: > >>> > >>> and this is after a few hours of running: > >>> > >>> Mem: 91M Active, 17G Inact, 3983M Wired, 1526M Cache, 3283M Buf, 155M Free > >>> Swap: 4096M Total, 828K Used, 4095M Free > >>> > >>> > >>> > >>> > >>> For comparison, here is the output of a 32GB FreeBSD 9.0/amd64 server, with Postgres 9.0.7 running since June 10, andis heavily pounded on 24x7. The data + indexes are about 240GB on disk. This server only runs postgres aside from thebasic system processes. > >>> > >>> > >>> Mem: 231M Active, 21G Inact, 3777M Wired, 1009M Cache, 3285M Buf, 191M Free > >>> Swap: 4096M Total, 272K Used, 4096M Free > >>> > >>> > >>> I agree with the conclusion that the shared memory segments are confusing the output of top. There are no memory leaks,and FreeBSD doesn't "lose" any memory. > >>> > >>> > >>> There are some scripts floating around that read values from sysctl vm.stats.vm and format them nicely to tell youhow much memory is used up and free. Try the one referenced here: http://www.cyberciti.biz/faq/freebsd-command-to-get-ram-information/ > >>> > >>> > >>> > >>> > >>> > >>> - > >>> Achilleas Mantzios > >>> IT DEPT > >>> > >> > >> > >> > > - > > Achilleas Mantzios > > IT DEPT > > > > > - Achilleas Mantzios IT DEPT