Thread: oom_killer

oom_killer

From
Tory M Blue
Date:
Is there anyone that could help me understand why all of a sudden with
no noticeable change in data, no change in hardware, no change in OS,
I'm seeing postmaster getting killed by oom_killer?

The dmesg shows that swap has not been touched free and total are the
same, so this system is not running out of total memory per say.

I keep thinking it's something to do with lowmem vs highmem 32bit vs
64 bit, but again no changes and I'm getting hit nightly on 2
different servers (running slon, so switched over and same thing, even
disabled memory over commit and still got nailed.

Is there anyone familiar with this or could take a look at the dmesg
output (off list) and decipher it for me?

this is a Fedora 12 system, 2.6.32.23-170. I've been reading and
appears this is yet another fedora bug, but so far I have not found
any concrete evidence on how to fix it.

Fedora 12
32gig memory, 8 proc
postgres 8.4.4, slony 1.20
5 gigs of swap (never hit it!)

Thanks
Tory

Re: oom_killer

From
yoshi watanabe
Date:
Funny concidence, I was just reading up a blog post on postgres an OOM killer.

http://gentooexperimental.org/~patrick/weblog/archives/2011-04.html#e2011-04-20T21_58_37.txt

Hope this helps.

2011/4/21 Tory M Blue <tmblue@gmail.com>:
> Is there anyone that could help me understand why all of a sudden with
> no noticeable change in data, no change in hardware, no change in OS,
> I'm seeing postmaster getting killed by oom_killer?
>
> The dmesg shows that swap has not been touched free and total are the
> same, so this system is not running out of total memory per say.
>
> I keep thinking it's something to do with lowmem vs highmem 32bit vs
> 64 bit, but again no changes and I'm getting hit nightly on 2
> different servers (running slon, so switched over and same thing, even
> disabled memory over commit and still got nailed.
>
> Is there anyone familiar with this or could take a look at the dmesg
> output (off list) and decipher it for me?
>
> this is a Fedora 12 system, 2.6.32.23-170. I've been reading and
> appears this is yet another fedora bug, but so far I have not found
> any concrete evidence on how to fix it.
>
> Fedora 12
> 32gig memory, 8 proc
> postgres 8.4.4, slony 1.20
> 5 gigs of swap (never hit it!)
>
> Thanks
> Tory
>
> --
> Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-performance
>

Re: oom_killer

From
Stephen Frost
Date:
* Tory M Blue (tmblue@gmail.com) wrote:
> Is there anyone that could help me understand why all of a sudden with
> no noticeable change in data, no change in hardware, no change in OS,
> I'm seeing postmaster getting killed by oom_killer?

You would really be best off just turning off the oom_killer..  Of
course, you should probably also figure out what process is actually
chewing through your memory to the point that the OOM killer is getting
run.

> The dmesg shows that swap has not been touched free and total are the
> same, so this system is not running out of total memory per say.

There's probably something else that's trying to grab all the memory and
then tries to use it and PG ends up getting nailed because the kernel
over-attributes memory to it.  You should be looking for that other
process..

    Thanks,

        Stephen

Attachment

Re: oom_killer

From
Claudio Freire
Date:
On Thu, Apr 21, 2011 at 2:48 PM, Stephen Frost <sfrost@snowman.net> wrote:
>
> There's probably something else that's trying to grab all the memory and
> then tries to use it and PG ends up getting nailed because the kernel
> over-attributes memory to it.  You should be looking for that other
> process..

Not only that, you probably should set up your oom killer not to kill
postmaster. Ever.

Re: oom_killer

From
Claudio Freire
Date:
On Thu, Apr 21, 2011 at 2:53 PM, Claudio Freire <klaussfreire@gmail.com> wrote:
> On Thu, Apr 21, 2011 at 2:48 PM, Stephen Frost <sfrost@snowman.net> wrote:
>>
>> There's probably something else that's trying to grab all the memory and
>> then tries to use it and PG ends up getting nailed because the kernel
>> over-attributes memory to it.  You should be looking for that other
>> process..
>
> Not only that, you probably should set up your oom killer not to kill
> postmaster. Ever.
>

Here: http://developer.postgresql.org/pgdocs/postgres/kernel-resources.html

Re: oom_killer

From
Merlin Moncure
Date:
On Thu, Apr 21, 2011 at 3:28 AM, Tory M Blue <tmblue@gmail.com> wrote:
> Is there anyone that could help me understand why all of a sudden with
> no noticeable change in data, no change in hardware, no change in OS,
> I'm seeing postmaster getting killed by oom_killer?
>
> The dmesg shows that swap has not been touched free and total are the
> same, so this system is not running out of total memory per say.
>
> I keep thinking it's something to do with lowmem vs highmem 32bit vs
> 64 bit, but again no changes and I'm getting hit nightly on 2
> different servers (running slon, so switched over and same thing, even
> disabled memory over commit and still got nailed.
>
> Is there anyone familiar with this or could take a look at the dmesg
> output (off list) and decipher it for me?
>
> this is a Fedora 12 system, 2.6.32.23-170. I've been reading and
> appears this is yet another fedora bug, but so far I have not found
> any concrete evidence on how to fix it.
>
> Fedora 12
> 32gig memory, 8 proc
> postgres 8.4.4, slony 1.20
> 5 gigs of swap (never hit it!)

curious: using 32/64 bit postgres? what are your postgresql.conf
memory settings?

merlin

Re: oom_killer

From
Tory M Blue
Date:
On Thu, Apr 21, 2011 at 7:27 AM, Merlin Moncure <mmoncure@gmail.com> wrote:
> On Thu, Apr 21, 2011 at 3:28 AM, Tory M Blue <tmblue@gmail.com> wrote:

>> Fedora 12
>> 32gig memory, 8 proc
>> postgres 8.4.4, slony 1.20
>> 5 gigs of swap (never hit it!)
>
> curious: using 32/64 bit postgres? what are your postgresql.conf
> memory settings?
>
> merlin
>

32bit
32gb
PAE kernel

# - Checkpoints -
checkpoint_segments = 100
max_connections = 300
shared_buffers = 2500MB       # min 128kB or max_connections*16kB
max_prepared_transactions = 0
work_mem = 100MB
maintenance_work_mem = 128MB
fsync = on

thanks
Tory

Re: oom_killer

From
Tory M Blue
Date:
On Thu, Apr 21, 2011 at 5:53 AM, Claudio Freire <klaussfreire@gmail.com> wrote:
> On Thu, Apr 21, 2011 at 2:48 PM, Stephen Frost <sfrost@snowman.net> wrote:
>>
>> There's probably something else that's trying to grab all the memory and
>> then tries to use it and PG ends up getting nailed because the kernel
>> over-attributes memory to it.  You should be looking for that other
>> process..
>
> Not only that, you probably should set up your oom killer not to kill
> postmaster. Ever.

Ya did that last night setting it to a -17 ya.

and to the other user stating I should disable oom_killer all together,

Ya of setting vm.overcommit to 2 and the ratio to 0 doesn't disable
it, I don't know what else to do. out of memory is out of memory, but
if swap is not being touched, I can't tell you what the heck this
fedora team is doing/thinking

Tory

Re: oom_killer

From
Claudio Freire
Date:
On Thu, Apr 21, 2011 at 5:50 PM, Tory M Blue <tmblue@gmail.com> wrote:
> # - Checkpoints -
> checkpoint_segments = 100
> max_connections = 300
> shared_buffers = 2500MB       # min 128kB or max_connections*16kB
> max_prepared_transactions = 0
> work_mem = 100MB
> maintenance_work_mem = 128MB
> fsync = on

That's an unrealistic setting for a 32-bit system, which can only
address 3GB of memory per process.

You take away 2500MB for shared buffers, that leaves you only 500M for
data, some of which is code.

There's no way PG can operate with 100MB work_mem llike that.

Either decrease shared_buffers, or get a 64-bit system.

Re: oom_killer

From
Tory M Blue
Date:
On Thu, Apr 21, 2011 at 8:57 AM, Claudio Freire <klaussfreire@gmail.com> wrote:
> On Thu, Apr 21, 2011 at 5:50 PM, Tory M Blue <tmblue@gmail.com> wrote:
>> # - Checkpoints -
>> checkpoint_segments = 100
>> max_connections = 300
>> shared_buffers = 2500MB       # min 128kB or max_connections*16kB
>> max_prepared_transactions = 0
>> work_mem = 100MB
>> maintenance_work_mem = 128MB
>> fsync = on
>
> That's an unrealistic setting for a 32-bit system, which can only
> address 3GB of memory per process.
>
> You take away 2500MB for shared buffers, that leaves you only 500M for
> data, some of which is code.
>
> There's no way PG can operate with 100MB work_mem llike that.
>
> Either decrease shared_buffers, or get a 64-bit system.

While I don't mind the occasional slap of reality. This configuration
has run for 4+ years. It's possible that as many other components each
fedora release is worse then the priors.

The Os has changed 170 days ago from fc6 to f12, but the postgres
configuration has been the same, and umm no way it can operate, is so
black and white, especially when it has ran performed well with a
decent sized data set for over 4 years.

This is not the first time I've posted configs to this list over the
last few years and not once has anyone pointed this shortcoming out or
said this will never work.

While i'm still a newb when it comes to postgres performance tuning, I
don't generally see things in black and white. And again zero swap is
being used but oom_killer is being called??

But if I remove

Re: oom_killer

From
Claudio Freire
Date:
On Thu, Apr 21, 2011 at 6:15 PM, Tory M Blue <tmblue@gmail.com> wrote:
> While I don't mind the occasional slap of reality. This configuration
> has run for 4+ years. It's possible that as many other components each
> fedora release is worse then the priors.

I'd say you've been lucky.
You must be running overnight report queries that didn't run before,
and that require more sorting memory than usual. Or... I dunno... but
something did change.

Re: oom_killer

From
Scott Marlowe
Date:
On Thu, Apr 21, 2011 at 11:15 AM, Tory M Blue <tmblue@gmail.com> wrote:

> While I don't mind the occasional slap of reality. This configuration
> has run for 4+ years. It's possible that as many other components each
> fedora release is worse then the priors.

How many of those 300 max connections do you generally use?  If you've
always used a handful, or you've used more but they weren't memory
hungry then you've been lucky.

work_mem is how much memory postgresql can allocate PER sort or hash
type operation.  Each connection can do that more than once.  A
complex query can do it dozens of times.  Can you see that going from
20 to 200 connections and increasing complexity can result in memory
usage going from a few megabytes to something like 200 connections *
100Megabytes per sort * 3 sorts = 60Gigabytes.

> The Os has changed 170 days ago from fc6 to f12, but the postgres
> configuration has been the same, and umm no way it can operate, is so
> black and white, especially when it has ran performed well with a
> decent sized data set for over 4 years.

Just because you've been walking around with a gun pointing at your
head without it going off does not mean walking around with a gun
pointing at your head is a good idea.

Re: oom_killer

From
Tory M Blue
Date:
On Thu, Apr 21, 2011 at 1:04 PM, Scott Marlowe <scott.marlowe@gmail.com> wrote:
> On Thu, Apr 21, 2011 at 11:15 AM, Tory M Blue <tmblue@gmail.com> wrote:
>
>> While I don't mind the occasional slap of reality. This configuration
>> has run for 4+ years. It's possible that as many other components each
>> fedora release is worse then the priors.
>
> How many of those 300 max connections do you generally use?  If you've
> always used a handful, or you've used more but they weren't memory
> hungry then you've been lucky.

max of 45

> work_mem is how much memory postgresql can allocate PER sort or hash
> type operation.  Each connection can do that more than once.  A
> complex query can do it dozens of times.  Can you see that going from
> 20 to 200 connections and increasing complexity can result in memory
> usage going from a few megabytes to something like 200 connections *
> 100Megabytes per sort * 3 sorts = 60Gigabytes.
>
>> The Os has changed 170 days ago from fc6 to f12, but the postgres
>> configuration has been the same, and umm no way it can operate, is so
>> black and white, especially when it has ran performed well with a
>> decent sized data set for over 4 years.
>
> Just because you've been walking around with a gun pointing at your
> head without it going off does not mean walking around with a gun
> pointing at your head is a good idea.


Yes that is what I gathered. It's good information and I'm always open
to a smack if I learn something, which in this case I did.

We were already working on moving to 64bit, but again the oom_killer
popping up without the system even attempting to use swap is what has
caused me some pause.

Thanks again
Tory

Re: oom_killer

From
J Sisson
Date:
On Thu, Apr 21, 2011 at 3:04 PM, Scott Marlowe <scott.marlowe@gmail.com> wrote:
> Just because you've been walking around with a gun pointing at your
> head without it going off does not mean walking around with a gun
> pointing at your head is a good idea.

+1

Re: oom_killer

From
Scott Marlowe
Date:
On Thu, Apr 21, 2011 at 3:08 PM, Tory M Blue <tmblue@gmail.com> wrote:
> On Thu, Apr 21, 2011 at 1:04 PM, Scott Marlowe <scott.marlowe@gmail.com> wrote:
>> On Thu, Apr 21, 2011 at 11:15 AM, Tory M Blue <tmblue@gmail.com> wrote:
>>
>>> While I don't mind the occasional slap of reality. This configuration
>>> has run for 4+ years. It's possible that as many other components each
>>> fedora release is worse then the priors.
>>
>> How many of those 300 max connections do you generally use?  If you've
>> always used a handful, or you've used more but they weren't memory
>> hungry then you've been lucky.
>
> max of 45
>
>> work_mem is how much memory postgresql can allocate PER sort or hash
>> type operation.  Each connection can do that more than once.  A
>> complex query can do it dozens of times.  Can you see that going from
>> 20 to 200 connections and increasing complexity can result in memory
>> usage going from a few megabytes to something like 200 connections *
>> 100Megabytes per sort * 3 sorts = 60Gigabytes.
>>
>>> The Os has changed 170 days ago from fc6 to f12, but the postgres
>>> configuration has been the same, and umm no way it can operate, is so
>>> black and white, especially when it has ran performed well with a
>>> decent sized data set for over 4 years.
>>
>> Just because you've been walking around with a gun pointing at your
>> head without it going off does not mean walking around with a gun
>> pointing at your head is a good idea.
>
>
> Yes that is what I gathered. It's good information and I'm always open
> to a smack if I learn something, which in this case I did.
>
> We were already working on moving to 64bit, but again the oom_killer
> popping up without the system even attempting to use swap is what has
> caused me some pause.

I think this might have been the 32 bit address space biting you.  But
that's just a guess.  Or the OS was running out of something other
than just plain memory, like file handles or something.  But I'm not
that familiar with OOM killer as it's one of the things I tend to shut
off when building a pg server.  I also turn off swap and zone_reclaim
mode.

Re: oom_killer

From
Merlin Moncure
Date:
On Thu, Apr 21, 2011 at 3:08 PM, Tory M Blue <tmblue@gmail.com> wrote:
> On Thu, Apr 21, 2011 at 1:04 PM, Scott Marlowe <scott.marlowe@gmail.com> wrote:
>> On Thu, Apr 21, 2011 at 11:15 AM, Tory M Blue <tmblue@gmail.com> wrote:
>>
>>> While I don't mind the occasional slap of reality. This configuration
>>> has run for 4+ years. It's possible that as many other components each
>>> fedora release is worse then the priors.
>>
>> How many of those 300 max connections do you generally use?  If you've
>> always used a handful, or you've used more but they weren't memory
>> hungry then you've been lucky.
>
> max of 45
>
>> work_mem is how much memory postgresql can allocate PER sort or hash
>> type operation.  Each connection can do that more than once.  A
>> complex query can do it dozens of times.  Can you see that going from
>> 20 to 200 connections and increasing complexity can result in memory
>> usage going from a few megabytes to something like 200 connections *
>> 100Megabytes per sort * 3 sorts = 60Gigabytes.
>>
>>> The Os has changed 170 days ago from fc6 to f12, but the postgres
>>> configuration has been the same, and umm no way it can operate, is so
>>> black and white, especially when it has ran performed well with a
>>> decent sized data set for over 4 years.
>>
>> Just because you've been walking around with a gun pointing at your
>> head without it going off does not mean walking around with a gun
>> pointing at your head is a good idea.
>
>
> Yes that is what I gathered. It's good information and I'm always open
> to a smack if I learn something, which in this case I did.
>
> We were already working on moving to 64bit, but again the oom_killer
> popping up without the system even attempting to use swap is what has
> caused me some pause.

Your shared_buffers is way way to high...you have dangerously
oversubscribed this system.  I would consider dropping down to
256-512mb.  Yeah, you have PAE but that only helps so much.  Your
server can only address so much memory and you allocated a huge chunk
of it right off the bat.

Also, you might want to consider connection pooler to keep your
#backends down, especially if you need to keep work_mem high.

merlin

Re: oom_killer

From
Cédric Villemain
Date:
2011/4/21 Tory M Blue <tmblue@gmail.com>:
> On Thu, Apr 21, 2011 at 7:27 AM, Merlin Moncure <mmoncure@gmail.com> wrote:
>> On Thu, Apr 21, 2011 at 3:28 AM, Tory M Blue <tmblue@gmail.com> wrote:
>
>>> Fedora 12
>>> 32gig memory, 8 proc
>>> postgres 8.4.4, slony 1.20
>>> 5 gigs of swap (never hit it!)
>>
>> curious: using 32/64 bit postgres? what are your postgresql.conf
>> memory settings?
>>
>> merlin
>>
>
> 32bit
> 32gb
> PAE kernel
>
> # - Checkpoints -
> checkpoint_segments = 100
> max_connections = 300
> shared_buffers = 2500MB       # min 128kB or max_connections*16kB
> max_prepared_transactions = 0
> work_mem = 100MB
> maintenance_work_mem = 128MB
> fsync = on
>

I didn't understand what value you set for vm.overcommit parameters.
Can you give it and the values in /proc/meminfo, the interesting one
are "Commit*" ?

If you have strict rules(overcommit=2), then your current kernel
config may need some love : the commit_limit is probably too low
because you have a small swap partition. One way is to change :
vm.overcommit_ratio.
By default it should be something like 21GB (0.5*32+5) of
commit_limit, and you probably want 32GB :)

Maybe you have some minor changes in your install or application usage
and you just hit the limit.

--
Cédric Villemain               2ndQuadrant
http://2ndQuadrant.fr/     PostgreSQL : Expertise, Formation et Support

Re: oom_killer

From
Tory M Blue
Date:
On Fri, Apr 22, 2011 at 4:03 AM, Cédric Villemain
<cedric.villemain.debian@gmail.com> wrote:
> 2011/4/21 Tory M Blue <tmblue@gmail.com>:
>> On Thu, Apr 21, 2011 at 7:27 AM, Merlin Moncure <mmoncure@gmail.com> wrote:
>>> On Thu, Apr 21, 2011 at 3:28 AM, Tory M Blue <tmblue@gmail.com> wrote:
>>
>>>> Fedora 12
>>>> 32gig memory, 8 proc
>>>> postgres 8.4.4, slony 1.20
>>>> 5 gigs of swap (never hit it!)
>>>
>>> curious: using 32/64 bit postgres? what are your postgresql.conf
>>> memory settings?
>>>
>>> merlin
>>>
>>
>> 32bit
>> 32gb
>> PAE kernel
>>
>> # - Checkpoints -
>> checkpoint_segments = 100
>> max_connections = 300
>> shared_buffers = 2500MB       # min 128kB or max_connections*16kB
>> max_prepared_transactions = 0
>> work_mem = 100MB
>> maintenance_work_mem = 128MB
>> fsync = on
>>
>
> I didn't understand what value you set for vm.overcommit parameters.
> Can you give it and the values in /proc/meminfo, the interesting one
> are "Commit*" ?
>
> If you have strict rules(overcommit=2), then your current kernel
> config may need some love : the commit_limit is probably too low
> because you have a small swap partition. One way is to change :
> vm.overcommit_ratio.
> By default it should be something like 21GB (0.5*32+5) of
> commit_limit, and you probably want 32GB :)
>
> Maybe you have some minor changes in your install or application usage
> and you just hit the limit.

Thanks Cedric

the sysctl vm's are

# 04/17/2011 to keep overcommit memory in check
vm.overcommit_memory = 2
vm.overcommit_ratio = 0

CommitLimit:     4128760 kB
Committed_AS:    2380408 kB


Ya I do think my swap space is biting us, (but again just starting to
grasp that my swap space which has not grown with the continued
addition of memory). I am just not starting to learn that the swap
does need to be properly sized whether it's being used or not. I
figured it would use the swap and it would run out, but sounds like
the system takes the size into consideration and just decides not to
use it.

I appreciate the totally no postgres responses with this.

Thanks
Tory

Re: oom_killer

From
"Kevin Grittner"
Date:
Tory M Blue <tmblue@gmail.com> wrote:

> I appreciate the totally no postgres responses with this.

I didn't understand that.  What do you mean?

-Kevin

Re: oom_killer

From
Tory M Blue
Date:
On Fri, Apr 22, 2011 at 9:34 AM, Kevin Grittner
<Kevin.Grittner@wicourts.gov> wrote:
> Tory M Blue <tmblue@gmail.com> wrote:
>
>> I appreciate the totally no postgres responses with this.
>
> I didn't understand that.  What do you mean?
>
> -Kevin

I meant that when starting to talk about kernel commit limits/ etc,
it's not really postgres centric, but you folks are still assisting me
with this.  So thanks, some could say take it to Linux kernels, even
though this is killing postgres.

Tory

Re: oom_killer

From
Cédric Villemain
Date:
2011/4/22 Tory M Blue <tmblue@gmail.com>:
> On Fri, Apr 22, 2011 at 4:03 AM, Cédric Villemain
> <cedric.villemain.debian@gmail.com> wrote:
>> 2011/4/21 Tory M Blue <tmblue@gmail.com>:
>>> On Thu, Apr 21, 2011 at 7:27 AM, Merlin Moncure <mmoncure@gmail.com> wrote:
>>>> On Thu, Apr 21, 2011 at 3:28 AM, Tory M Blue <tmblue@gmail.com> wrote:
>>>
>>>>> Fedora 12
>>>>> 32gig memory, 8 proc
>>>>> postgres 8.4.4, slony 1.20
>>>>> 5 gigs of swap (never hit it!)
>>>>
>>>> curious: using 32/64 bit postgres? what are your postgresql.conf
>>>> memory settings?
>>>>
>>>> merlin
>>>>
>>>
>>> 32bit
>>> 32gb
>>> PAE kernel
>>>
>>> # - Checkpoints -
>>> checkpoint_segments = 100
>>> max_connections = 300
>>> shared_buffers = 2500MB       # min 128kB or max_connections*16kB
>>> max_prepared_transactions = 0
>>> work_mem = 100MB
>>> maintenance_work_mem = 128MB
>>> fsync = on
>>>
>>
>> I didn't understand what value you set for vm.overcommit parameters.
>> Can you give it and the values in /proc/meminfo, the interesting one
>> are "Commit*" ?
>>
>> If you have strict rules(overcommit=2), then your current kernel
>> config may need some love : the commit_limit is probably too low
>> because you have a small swap partition. One way is to change :
>> vm.overcommit_ratio.
>> By default it should be something like 21GB (0.5*32+5) of
>> commit_limit, and you probably want 32GB :)
>>
>> Maybe you have some minor changes in your install or application usage
>> and you just hit the limit.
>
> Thanks Cedric
>
> the sysctl vm's are
>
> # 04/17/2011 to keep overcommit memory in check
> vm.overcommit_memory = 2
> vm.overcommit_ratio = 0
>
> CommitLimit:     4128760 kB
> Committed_AS:    2380408 kB

Are you sure it is a PAE kernel ? You look limited to 4GB.

I don't know atm if overcommit_ratio=0 has a special meaning, else I
would suggest to update it to something like 40% (the default), but
60% should still be safe (60% of 32GB + 5GB)

>
>
> Ya I do think my swap space is biting us, (but again just starting to
> grasp that my swap space which has not grown with the continued
> addition of memory). I am just not starting to learn that the swap
> does need to be properly sized whether it's being used or not. I
> figured it would use the swap and it would run out, but sounds like
> the system takes the size into consideration and just decides not to
> use it.
>
> I appreciate the totally no postgres responses with this.
>
> Thanks
> Tory
>



--
Cédric Villemain               2ndQuadrant
http://2ndQuadrant.fr/     PostgreSQL : Expertise, Formation et Support

Re: oom_killer

From
Cédric Villemain
Date:
2011/4/22 Cédric Villemain <cedric.villemain.debian@gmail.com>:
> 2011/4/22 Tory M Blue <tmblue@gmail.com>:
>> On Fri, Apr 22, 2011 at 4:03 AM, Cédric Villemain
>> <cedric.villemain.debian@gmail.com> wrote:
>>> 2011/4/21 Tory M Blue <tmblue@gmail.com>:
>>>> On Thu, Apr 21, 2011 at 7:27 AM, Merlin Moncure <mmoncure@gmail.com> wrote:
>>>>> On Thu, Apr 21, 2011 at 3:28 AM, Tory M Blue <tmblue@gmail.com> wrote:
>>>>
>>>>>> Fedora 12
>>>>>> 32gig memory, 8 proc
>>>>>> postgres 8.4.4, slony 1.20
>>>>>> 5 gigs of swap (never hit it!)
>>>>>
>>>>> curious: using 32/64 bit postgres? what are your postgresql.conf
>>>>> memory settings?
>>>>>
>>>>> merlin
>>>>>
>>>>
>>>> 32bit
>>>> 32gb
>>>> PAE kernel
>>>>
>>>> # - Checkpoints -
>>>> checkpoint_segments = 100
>>>> max_connections = 300
>>>> shared_buffers = 2500MB       # min 128kB or max_connections*16kB
>>>> max_prepared_transactions = 0
>>>> work_mem = 100MB
>>>> maintenance_work_mem = 128MB
>>>> fsync = on
>>>>
>>>
>>> I didn't understand what value you set for vm.overcommit parameters.
>>> Can you give it and the values in /proc/meminfo, the interesting one
>>> are "Commit*" ?
>>>
>>> If you have strict rules(overcommit=2), then your current kernel
>>> config may need some love : the commit_limit is probably too low
>>> because you have a small swap partition. One way is to change :
>>> vm.overcommit_ratio.
>>> By default it should be something like 21GB (0.5*32+5) of
>>> commit_limit, and you probably want 32GB :)
>>>
>>> Maybe you have some minor changes in your install or application usage
>>> and you just hit the limit.
>>
>> Thanks Cedric
>>
>> the sysctl vm's are
>>
>> # 04/17/2011 to keep overcommit memory in check
>> vm.overcommit_memory = 2
>> vm.overcommit_ratio = 0
>>
>> CommitLimit:     4128760 kB
>> Committed_AS:    2380408 kB
>
> Are you sure it is a PAE kernel ? You look limited to 4GB.
>
> I don't know atm if overcommit_ratio=0 has a special meaning, else I
> would suggest to update it to something like 40% (the default), but

default being 50 ...

> 60% should still be safe (60% of 32GB + 5GB)
>
>>
>>
>> Ya I do think my swap space is biting us, (but again just starting to
>> grasp that my swap space which has not grown with the continued
>> addition of memory). I am just not starting to learn that the swap
>> does need to be properly sized whether it's being used or not. I
>> figured it would use the swap and it would run out, but sounds like
>> the system takes the size into consideration and just decides not to
>> use it.
>>
>> I appreciate the totally no postgres responses with this.
>>
>> Thanks
>> Tory
>>
>
>
>
> --
> Cédric Villemain               2ndQuadrant
> http://2ndQuadrant.fr/     PostgreSQL : Expertise, Formation et Support
>



--
Cédric Villemain               2ndQuadrant
http://2ndQuadrant.fr/     PostgreSQL : Expertise, Formation et Support

Re: oom_killer

From
Tory M Blue
Date:
On Fri, Apr 22, 2011 at 9:46 AM, Cédric Villemain
<cedric.villemain.debian@gmail.com> wrote:
> 2011/4/22 Cédric Villemain <cedric.villemain.debian@gmail.com>:


>> Are you sure it is a PAE kernel ? You look limited to 4GB.
>>
>> I don't know atm if overcommit_ratio=0 has a special meaning, else I
>> would suggest to update it to something like 40% (the default), but
>
> default being 50 ...
>
>> 60% should still be safe (60% of 32GB + 5GB)

2.6.32.23-170.fc12.i686.PAE , so it says. Okay so instead of dropping
to 2-0 with the overcommit and ratio settings, I should look at
matching the ratio more to what I actually have in swap? Sorry but
knee jerk reaction when I got hit by the oom_killer twice.

Tory

Re: oom_killer

From
David Rees
Date:
On Thu, Apr 21, 2011 at 1:28 AM, Tory M Blue <tmblue@gmail.com> wrote:
> this is a Fedora 12 system, 2.6.32.23-170. I've been reading and
> appears this is yet another fedora bug, but so far I have not found
> any concrete evidence on how to fix it.

If it's a "fedora" bug, it's most likely related to the kernel where
the OOM-killer lives which really makes it more of a kernel bug than a
fedora bug as fedora kernels generally track upstream very closely.

Given that both the version of Fedora you're using is no longer
supported, at a minimum you should be running F-13 (or preferably F-14
since F-13 will lose maintenance in appx 2 months).  If you have to
stay on F-12 you might at least try building the latest
2.6.32-longterm kernel which is up to version 2.6.32.39.

All that said - have you tried tracking memory usage of the machine
leading up to OOM killer events?

-Dave

Re: oom_killer

From
Tory M Blue
Date:
On Fri, Apr 22, 2011 at 11:15 AM, David Rees <drees76@gmail.com> wrote:
> On Thu, Apr 21, 2011 at 1:28 AM, Tory M Blue <tmblue@gmail.com> wrote:
>> this is a Fedora 12 system, 2.6.32.23-170. I've been reading and
>> appears this is yet another fedora bug, but so far I have not found
>> any concrete evidence on how to fix it.
>
> If it's a "fedora" bug, it's most likely related to the kernel where
> the OOM-killer lives which really makes it more of a kernel bug than a
> fedora bug as fedora kernels generally track upstream very closely.
>
> Given that both the version of Fedora you're using is no longer
> supported, at a minimum you should be running F-13 (or preferably F-14
> since F-13 will lose maintenance in appx 2 months).  If you have to
> stay on F-12 you might at least try building the latest
> 2.6.32-longterm kernel which is up to version 2.6.32.39.
>
> All that said - have you tried tracking memory usage of the machine
> leading up to OOM killer events?
>

Thanks David and I have and in fact I do see spikes that would cause
my system to run out of memory, but one thing I'm struggling with is
my system always runs at the limit. It's the nature of linux to take
all the memory and manage it. The larger hurdle is why no swap is ever
used, it's there, but the system never uses it. even the oom killer
shows that I have the full 5gb of swap available, yet nothing is using
is. I want want want to see swap being used!

If I run a script to do a bunch of malocs and hold I can see the
system use up available memory then lay the smack down on my swap
before oom is invoked.

So I'm starting to think in the meantime, while I rebuild, I need to
make sure I've got my postgres/kernel params in a good place. my ratio
of 0 still allows oom_killer, but I've removed postgres from being
targeted by oom_killer now. I should still set the overcommit ratio
correct for my 32gb 4-5gb swap system, but having a hard time wrapping
my head around that setting.

Tory

Re: oom_killer

From
Tory M Blue
Date:
On Fri, Apr 22, 2011 at 9:45 AM, Cédric Villemain
<cedric.villemain.debian@gmail.com> wrote:

>> CommitLimit:     4128760 kB
>> Committed_AS:    2380408 kB
>
> Are you sure it is a PAE kernel ? You look limited to 4GB.

Figured that the Commitlimit is actually the size of swap, so on one
server it's 4gb and the other it's 5gb.

So still need to figure out with 32gig of ram and 4 to 5gig swap, what
my overcommit ratio should be.

Tory

Re: oom_killer

From
Cédric Villemain
Date:
2011/4/22 Tory M Blue <tmblue@gmail.com>:
> On Fri, Apr 22, 2011 at 9:45 AM, Cédric Villemain
> <cedric.villemain.debian@gmail.com> wrote:
>
>>> CommitLimit:     4128760 kB
>>> Committed_AS:    2380408 kB
>>
>> Are you sure it is a PAE kernel ? You look limited to 4GB.
>
> Figured that the Commitlimit is actually the size of swap, so on one
> server it's 4gb and the other it's 5gb.
>
> So still need to figure out with 32gig of ram and 4 to 5gig swap, what
> my overcommit ratio should be.

at least the default value of 50, probably more, up to you to adjust.
You should have something ok with 50, given that it used to work well
until now with 0 (so you'll have 21GB of commitable memory )

>
> Tory
>
> --
> Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-performance
>



--
Cédric Villemain               2ndQuadrant
http://2ndQuadrant.fr/     PostgreSQL : Expertise, Formation et Support

Re: oom_killer

From
Claudio Freire
Date:
On Fri, Apr 22, 2011 at 6:45 PM, Cédric Villemain
<cedric.villemain.debian@gmail.com> wrote:
> Are you sure it is a PAE kernel ? You look limited to 4GB.

If my memory/knowledge serves me right, PAE doesn't remove that limit.
PAE allows more processes, and they can use more memory together, but
one process alone has to live within an addressable range, and that is
still 4GB, mandated by the 32-bit addressable space when operating in
linear addressing mode.

But linux kernels usually reserve 1GB for kernel stuff (buffers and
that kind of stuff), so the addressable portion for processes is 3GB.

Take away 2.5GB of shared buffers, and you only leave 0.5G for general
data and code.

Really, lowering shared_buffers will probably be a solution. Moving to
64 bits would be a better one.

Re: oom_killer

From
Robert Haas
Date:
On Apr 22, 2011, at 2:22 PM, Tory M Blue <tmblue@gmail.com> wrote:
> Thanks David and I have and in fact I do see spikes that would cause
> my system to run out of memory, but one thing I'm struggling with is
> my system always runs at the limit. It's the nature of linux to take
> all the memory and manage it.

One thing to watch is the size of the filesystem cache. Generally as the system comes under memory pressure you will
seethe cache shrink. Not sure what is happening on your system, but typically when it gets down to some minimal size,
that'swhen the swapping starts. 

...Robert

Re: oom_killer

From
Tory M Blue
Date:
On Sat, Apr 23, 2011 at 12:24 PM, Robert Haas <robertmhaas@gmail.com> wrote:

> One thing to watch is the size of the filesystem cache. Generally as the system comes under memory pressure you will
seethe cache shrink. Not sure what is happening on your system, but typically when it gets down to some minimal size,
that'swhen the swapping starts. 
>
> ...Robert

Thanks everyone, I've tuned the system in the tune of overcommit 2 and
ratio of 80% this makes my commit look like:
CommitLimit:    31694880 kB
Committed_AS:    2372084 kB


So with 32G of system memory and 4gb cache so far it's running okay,
no ooms in the last 2 days and the DB is performing well again. I've
also dropped the shared buffers to 2gb, that gives me 1 gb for data
etc. I'll test with smaller 1.5gb if need be.

I've already started the 64bit process, I've got to test if slon will
replicate between a 32bit and 64 bit system, if the postgres/slon
versions are the same (slon being the key here). If this works, I will
be able to do the migration to 64bit that  much easier, if not well,
ya that changes the scheme a ton.

Thanks for the all the assistance in this, it's really appreciated

Tory