Thread: out of memory

out of memory

From
Fabricio
Date:

Hi

I have a dedicated database server with 16 GB of RAM.

and the oom-killer is killing my database

this is the output kernel:

postmaster invoked oom-killer: gfp_mask=0xd0, order=1, oomkilladj=0
Pid: 16667, comm: postmaster Tainted: G        W 2.6.27.6 #7
 [<c10510f9>] oom_kill_process+0x103/0x1d5
 [<c11c14cc>] security_capable+0x9/0xa
 [<c1050f0d>] badness+0x118/0x201
 [<c10513fc>] out_of_memory+0x231/0x265
 [<c1053300>] __alloc_pages_internal+0x322/0x447
 [<c106ac5e>] alloc_pages_current+0x59/0x89
 [<c1051c2c>] __get_free_pages+0xa/0x16
 [<c101f815>] copy_process+0xa2/0xeaa
 [<c1020683>] do_fork+0x66/0x2c3
 [<c1001a59>] sys_clone+0x32/0x36
 [<c100334a>] syscall_call+0x7/0xb
 =======================
Mem-Info:
Node 0 DMA per-cpu:
CPU    0: hi:    0, btch:   1 usd:   0
CPU    1: hi:    0, btch:   1 usd:   0
CPU    2: hi:    0, btch:   1 usd:   0
CPU    3: hi:    0, btch:   1 usd:   0
CPU    4: hi:    0, btch:   1 usd:   0
CPU    5: hi:    0, btch:   1 usd:   0
CPU    6: hi:    0, btch:   1 usd:   0
CPU    7: hi:    0, btch:   1 usd:   0
CPU    8: hi:    0, btch:   1 usd:   0
CPU    9: hi:    0, btch:   1 usd:   0
CPU   10: hi:    0, btch:   1 usd:   0
CPU   11: hi:    0, btch:   1 usd:   0
CPU   12: hi:    0, btch:   1 usd:   0
CPU   13: hi:    0, btch:   1 usd:   0
CPU   14: hi:    0, btch:   1 usd:   0
CPU   15: hi:    0, btch:   1 usd:   0
Node 0 Normal per-cpu:
CPU    0: hi:  186, btch:  31 usd:   0
CPU    1: hi:  186, btch:  31 usd:  30
CPU    2: hi:  186, btch:  31 usd:   0
CPU    3: hi:  186, btch:  31 usd:   0
CPU    4: hi:  186, btch:  31 usd:   0
CPU    5: hi:  186, btch:  31 usd:   0
CPU    6: hi:  186, btch:  31 usd:  26
CPU    7: hi:  186, btch:  31 usd:  26
CPU    8: hi:  186, btch:  31 usd:   0
CPU    9: hi:  186, btch:  31 usd:   0
CPU   10: hi:  186, btch:  31 usd:   0
CPU   11: hi:  186, btch:  31 usd:   0
CPU   12: hi:  186, btch:  31 usd:   0
CPU   13: hi:  186, btch:  31 usd:   0
CPU   14: hi:  186, btch:  31 usd:   0
CPU   15: hi:  186, btch:  31 usd:  25
Node 0 HighMem per-cpu:
CPU    0: hi:  186, btch:  31 usd:   0
CPU    1: hi:  186, btch:  31 usd:  30
CPU    2: hi:  186, btch:  31 usd:   0
CPU    3: hi:  186, btch:  31 usd:   0
CPU    4: hi:  186, btch:  31 usd:   0
CPU    5: hi:  186, btch:  31 usd: 167
CPU    6: hi:  186, btch:  31 usd:  25
CPU    7: hi:  186, btch:  31 usd:  19
CPU    8: hi:  186, btch:  31 usd:   0
CPU    9: hi:  186, btch:  31 usd:   0
CPU   10: hi:  186, btch:  31 usd:   0
CPU   11: hi:  186, btch:  31 usd:   0
CPU   12: hi:  186, btch:  31 usd:   0
CPU   13: hi:  186, btch:  31 usd:   0
CPU   14: hi:  186, btch:  31 usd:   0
CPU   15: hi:  186, btch:  31 usd:  23
Active:1573667 inactive:609150 dirty:46 writeback:107 unstable:0
 free:1750894 slab:31196 mapped:538832 pagetables:143359 bounce:0
Node 0 DMA free:3540kB min:64kB low:80kB high:96kB active:0kB inactive:0kB prese                                                                             nt:15836kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 873 16095 16095
Node 0 Normal free:28132kB min:3744kB low:4680kB high:5616kB active:840kB inacti                                                                             ve:892kB present:894080kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 121778 121778
Node 0 HighMem free:6971904kB min:512kB low:16836kB high:33160kB active:6293828k                                                                             B inactive:2435708kB present:15587608kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 0 DMA: 113*4kB 11*8kB 1*16kB 1*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB                                                                              1*2048kB 0*4096kB = 3660kB
Node 0 Normal: 6429*4kB 147*8kB 9*16kB 2*32kB 2*64kB 1*128kB 0*256kB 1*512kB 1*1                                                                             024kB 0*2048kB 0*4096kB = 28892kB
Node 0 HighMem: 690211*4kB 364384*8kB 72902*16kB 3650*32kB 17*64kB 50*128kB 5*25                                                                             6kB 2*512kB 1*1024kB 0*2048kB 1*4096kB = 6974060kB
1920296 total pagecache pages
0 pages in swap cache
Swap cache stats: add 913280, delete 913280, find 207387/229468
Free swap  = 1754452kB
Total swap = 3904552kB
4390911 pages RAM
4161535 pages HighMem
268987 pages reserved
6066507 pages shared
1384540 pages non-shared
Out of memory: kill process 13279 (postmaster) score 546058 or a child
Killed process 13279 (postmaster)


This is the output of free -m before and after the kill

free -m
             total       used       free     shared    buffers     cached
Mem:         16101       9302       6799          0          3       7521
-/+ buffers/cache:       1776      14324
Swap:         3813       2099       1713


free -m
             total       used       free     shared    buffers     cached
Mem:         16101       5606      10494          0          7       5410
-/+ buffers/cache:        188      15912
Swap:         3813       2099       1713


the sysctl.conf  with
kernel.shmmax = 10737418240
vm.overcommit_memory = 2

I do not what why the oom-killer kill the postgres if there are available memory.

I would appreciate any help

Regars...


No olvides consultar tus correos desde la ventana principal de Messenger

Re: out of memory

From
Vladimir Rusinov
Date:


On Thu, Mar 12, 2009 at 9:17 PM, Fabricio <fabrixio1@hotmail.com> wrote:

Hi

I have a dedicated database server with 16 GB of RAM.

and the oom-killer is killing my database

What kernel and architecture are you running? Show your `uname -a` please.



--
Vladimir Rusinov
http://greenmice.info/

Re: out of memory

From
Fabricio
Date:
 
Hello...
 
Linux 2.6.27.6 #7 SMP Sun Nov 16 00:48:35 MST 2008 i686 i686 i386 GNU/Linux
Slackware 11 and Postgres 8.1.15
 
thanks...


Date: Fri, 13 Mar 2009 00:03:16 +0300
Subject: Re: [ADMIN] out of memory
From: vladimir@greenmice.info
To: fabrixio1@hotmail.com
CC: pgsql-admin@postgresql.org



On Thu, Mar 12, 2009 at 9:17 PM, Fabricio <fabrixio1@hotmail.com> wrote:

Hi

I have a dedicated database server with 16 GB of RAM.

and the oom-killer is killing my database

What kernel and architecture are you running? Show your `uname -a` please.



--
Vladimir Rusinov
http://greenmice.info/


Personaliza, comparte y comunícate como nunca antes en Messenger

Re: out of memory

From
Fabricio
Date:

Hi

there's no comments?

Fabricio


From: fabrixio1@hotmail.com
To: pgsql-admin@postgresql.org
Subject: [ADMIN] out of memory
Date: Thu, 12 Mar 2009 11:17:52 -0700

.ExternalClass .EC_hmmessage P {padding:0px;} .ExternalClass body.EC_hmmessage {font-size:10pt;font-family:Verdana;}
Hi

I have a dedicated database server with 16 GB of RAM.
Linux 2.6.27.6 #7 SMP Sun Nov 16 00:48:35 MST 2008 i686 i686 i386 GNU/Linux
Slackware 11 and Postgres 8.1.15

and the oom-killer is killing my database

this is the output kernel:

postmaster invoked oom-killer: gfp_mask=0xd0, order=1, oomkilladj=0
Pid: 16667, comm: postmaster Tainted: G        W 2.6.27.6 #7
 [<c10510f9>] oom_kill_process+0x103/0x1d5
 [<c11c14cc>] security_capable+0x9/0xa
 [<c1050f0d>] badness+0x118/0x201
 [<c10513fc>] out_of_memory+0x231/0x265
 [<c1053300>] __alloc_pages_internal+0x322/0x447
 [<c106ac5e>] alloc_pages_current+0x59/0x89
 [<c1051c2c>] __get_free_pages+0xa/0x16
 [<c101f815>] copy_process+0xa2/0xeaa
 [<c1020683>] do_fork+0x66/0x2c3
 [<c1001a59>] sys_clone+0x32/0x36
 [<c100334a>] syscall_call+0x7/0xb
 =======================
Mem-Info:
Node 0 DMA per-cpu:
CPU    0: hi:    0, btch:   1 usd:   0
CPU    1: hi:    0, btch:   1 usd:   0
CPU    2: hi:    0, btch:   1 usd:   0
CPU    3: hi:    0, btch:   1 usd:   0
CPU    4: hi:    0, btch:   1 usd:   0
CPU    5: hi:    0, btch:   1 usd:   0
CPU    6: hi:    0, btch:   1 usd:   0
CPU    7: hi:    0, btch:   1 usd:   0
CPU    8: hi:    0, btch:   1 usd:   0
CPU    9: hi:    0, btch:   1 usd:   0
CPU   10: hi:    0, btch:   1 usd:   0
CPU   11: hi:    0, btch:   1 usd:   0
CPU   12: hi:    0, btch:   1 usd:   0
CPU   13: hi:    0, btch:   1 usd:   0
CPU   14: hi:    0, btch:   1 usd:   0
CPU   15: hi:    0, btch:   1 usd:   0
Node 0 Normal per-cpu:
CPU    0: hi:  186, btch:  31 usd:   0
CPU    1: hi:  186, btch:  31 usd:  30
CPU    2: hi:  186, btch:  31 usd:   0
CPU    3: hi:  186, btch:  31 usd:   0
CPU    4: hi:  186, btch:  31 usd:   0
CPU    5: hi:  186, btch:  31 usd:   0
CPU    6: hi:  186, btch:  31 usd:  26
CPU    7: hi:  186, btch:  31 usd:  26
CPU    8: hi:  186, btch:  31 usd:   0
CPU    9: hi:  186, btch:  31 usd:   0
CPU   10: hi:  186, btch:  31 usd:   0
CPU   11: hi:  186, btch:  31 usd:   0
CPU   12: hi:  186, btch:  31 usd:   0
CPU   13: hi:  186, btch:  31 usd:   0
CPU   14: hi:  186, btch:  31 usd:   0
CPU   15: hi:  186, btch:  31 usd:  25
Node 0 HighMem per-cpu:
CPU    0: hi:  186, btch:  31 usd:   0
CPU    1: hi:  186, btch:  31 usd:  30
CPU    2: hi:  186, btch:  31 usd:   0
CPU    3: hi:  186, btch:  31 usd:   0
CPU    4: hi:  186, btch:  31 usd:   0
CPU    5: hi:  186, btch:  31 usd: 167
CPU    6: hi:  186, btch:  31 usd:  25
CPU    7: hi:  186, btch:  31 usd:  19
CPU    8: hi:  186, btch:  31 usd:   0
CPU    9: hi:  186, btch:  31 usd:   0
CPU   10: hi:  186, btch:  31 usd:   0
CPU   11: hi:  186, btch:  31 usd:   0
CPU   12: hi:  186, btch:  31 usd:   0
CPU   13: hi:  186, btch:  31 usd:   0
CPU   14: hi:  186, btch:  31 usd:   0
CPU   15: hi:  186, btch:  31 usd:  23
Active:1573667 inactive:609150 dirty:46 writeback:107 unstable:0
 free:1750894 slab:31196 mapped:538832 pagetables:143359 bounce:0
Node 0 DMA free:3540kB min:64kB low:80kB high:96kB active:0kB inactive:0kB prese                                                                             nt:15836kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 873 16095 16095
Node 0 Normal free:28132kB min:3744kB low:4680kB high:5616kB active:840kB inacti                                                                             ve:892kB present:894080kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 121778 121778
Node 0 HighMem free:6971904kB min:512kB low:16836kB high:33160kB active:6293828k                                                                             B inactive:2435708kB present:15587608kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 0 DMA: 113*4kB 11*8kB 1*16kB 1*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB                                                                              1*2048kB 0*4096kB = 3660kB
Node 0 Normal: 6429*4kB 147*8kB 9*16kB 2*32kB 2*64kB 1*128kB 0*256kB 1*512kB 1*1                                                                             024kB 0*2048kB 0*4096kB = 28892kB
Node 0 HighMem: 690211*4kB 364384*8kB 72902*16kB 3650*32kB 17*64kB 50*128kB 5*25                                                                             6kB 2*512kB 1*1024kB 0*2048kB 1*4096kB = 6974060kB
1920296 total pagecache pages
0 pages in swap cache
Swap cache stats: add 913280, delete 913280, find 207387/229468
Free swap  = 1754452kB
Total swap = 3904552kB
4390911 pages RAM
4161535 pages HighMem
268987 pages reserved
6066507 pages shared
1384540 pages non-shared
Out of memory: kill process 13279 (postmaster) score 546058 or a child
Killed process 13279 (postmaster)


This is the output of free -m before and after the kill

free -m
             total       used       free     shared    buffers     cached
Mem:         16101       9302       6799          0          3       7521
-/+ buffers/cache:       1776      14324
Swap:         3813       2099       1713


free -m
             total       used       free     shared    buffers     cached
Mem:         16101       5606      10494          0          7       5410
-/+ buffers/cache:        188      15912
Swap:         3813       2099       1713


the sysctl.conf  with
kernel.shmmax = 10737418240
vm.overcommit_memory = 2

I do not what why the oom-killer kill the postgres if there are available memory.

I would appreciate any help

Regars...


No olvides consultar tus correos desde la ventana principal de Messenger

Si eres de los que usa Messenger no solo para chatear. esto te va a interesar

Re: out of memory

From
Alvaro Herrera
Date:
Fabricio wrote:
>
>
> Hi
>
> I have a dedicated database server with 16 GB of RAM.
>
> and the oom-killer is killing my database

Try disabling memory overcommit; see 17.4.3 here:
http://www.postgresql.org/docs/8.3/interactive/kernel-resources.html

--
Alvaro Herrera                                http://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.

Re: out of memory

From
Vladimir Rusinov
Date:


On Fri, Mar 13, 2009 at 11:15 PM, Fabricio <fabrixio1@hotmail.com> wrote:
 
Hello...
 
Linux 2.6.27.6 #7 SMP Sun Nov 16 00:48:35 MST 2008 i686 i686 i386 GNU/Linux
Slackware 11 and Postgres 8.1.15


Go download 64-bit OS.

--
Vladimir Rusinov
http://greenmice.info/

Re: out of memory

From
Fabricio
Date:

Thanks for answer Alvaro
 
Already i have memory overcommit disabled.
I've always had with the value 2

vm.overcommit_memory = 2

Fabricio

> Date: Wed, 18 Mar 2009 20:28:02 -0400
> From: alvherre@commandprompt.com
> To: fabrixio1@hotmail.com
> CC: pgsql-admin@postgresql.org
> Subject: Re: [ADMIN] out of memory
>
> Fabricio wrote:
> >
> >
> > Hi
> >
> > I have a dedicated database server with 16 GB of RAM.
> >
> > and the oom-killer is killing my database
>
> Try disabling memory overcommit; see 17.4.3 here:
> http://www.postgresql.org/docs/8.3/interactive/kernel-resources.html
>
> --
> Alvaro Herrera http://www.CommandPrompt.com/
> The PostgreSQL Company - Command Prompt, Inc.


Personaliza, comparte y comunícate como nunca antes en Messenger

Re: out of memory

From
Fabricio
Date:


Hi

Some one know why this is happening?

I change the OS to 64 bits and now the oom-killer not hapend but Postgres is still showing out of memory

Linux SERVER 2.6.18-92.el5 #1 SMP Tue Apr 29 13:16:15 EDT 2008 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 5.2 (Tikanga)
PostgreSQL 8.1.15
32GB RAM

My postgresql.conf:
# - Memory -

shared_buffers = 1048576                        # min 16 or max_connections*2, 8KB each
temp_buffers = 1024                     # min 100, 8KB each
max_prepared_transactions = 20         # can be 0 or more
# note: increasing max_prepared_transactions costs ~600 bytes of shared memory
# per transaction slot, plus lock space (see max_locks_per_transaction).
work_mem = 1024                 # min 64, size in KB
maintenance_work_mem = 65536            # min 1024, size in KB
max_stack_depth = 2048                  # min 100, size in KB



kernel messages:
Jul 31 11:50:08 SERVER kernel: postmaster[7686]: segfault at 00007fff3feb1bb0 rip 00002b2f7e17e1a8 rsp 00007fff3feb1b90 error 6
Jul 31 15:41:55 SERVER kernel: postmaster[4737]: segfault at 00007fff3feb1bb0 rip 00002b2f7e1851a8 rsp 00007fff3feb1b90 error 6

PostgreSQL log:
<2009-07-31 15:41:55 MDT    7253 > LOG:  could not fork new process for connection: Cannot allocate memory
<2009-07-31 15:41:55 MDT    7253 > LOG:  could not fork new process for connection: Cannot allocate memory
<2009-07-31 15:41:55 MDT 10.27.41.74(2606) aforeglobal sysaforeglobal 7423 startup> FATAL:  out of memory
<2009-07-31 15:41:55 MDT    7253 > LOG:  could not fork new process for connection: Cannot allocate memory
<2009-07-31 15:41:55 MDT    7253 > LOG:  could not fork new process for connection: Cannot allocate memory
<2009-07-31 15:41:55 MDT 10.27.36.219(3859) db user 7424 startup> FATAL:  out of memory
<2009-07-31 15:41:55 MDT 10.20.216.82(1966) db user 7431 startup> FATAL:  out of memory
TopMemoryContext: 164432 total in 6 blocks; 5368 free (1 chunks); 159064 used
MdSmgr: 0 total in 0 blocks; 0 free (0 chunks); 0 used
LockTable (locallock hash): 8192 total in 1 blocks; 3744 free (0 chunks); 4448 used
Timezones: 52560 total in 2 blocks; 3744 free (0 chunks); 48816 used
ErrorContext: 8192 total in 1 blocks; 8160 free (4 chunks); 32 used
<2009-07-31 15:41:55 MDT 10.33.128.38(4458) db user 7434 startup> FATAL:  out of memory
<2009-07-31 15:41:55 MDT 10.33.128.38(4458) db user 7434 startup> DETAIL:  Failed on request of size 24000.
TopMemoryContext: 164432 total in 6 blocks; 5368 free (1 chunks); 159064 used
MdSmgr: 0 total in 0 blocks; 0 free (0 chunks); 0 used
LockTable (locallock hash): 8192 total in 1 blocks; 3744 free (0 chunks); 4448 used
Timezones: 52560 total in 2 blocks; 3744 free (0 chunks); 48816 used
ErrorContext: 8192 total in 1 blocks; 8160 free (4 chunks); 32 used
<2009-07-31 15:41:55 MDT 10.33.128.38(4459) db user 7435 startup> FATAL:  out of memory
<2009-07-31 15:41:55 MDT 10.33.128.38(4459) db user 7435 startup> DETAIL:  Failed on request of size 24000.
TopMemoryContext: 164432 total in 6 blocks; 5368 free (1 chunks); 159064 used
MdSmgr: 0 total in 0 blocks; 0 free (0 chunks); 0 used
LockTable (locallock hash): 8192 total in 1 blocks; 3744 free (0 chunks); 4448 used
Timezones: 52560 total in 2 blocks; 3744 free (0 chunks); 48816 used
ErrorContext: 8192 total in 1 blocks; 8160 free (4 chunks); 32 used
<2009-07-31 15:41:55 MDT 10.33.128.38(4460) db user 7436 startup> FATAL:  out of memory
<2009-07-31 15:41:55 MDT 10.33.128.38(4460) db user 7436 startup> DETAIL:  Failed on request of size 24000.
<2009-07-31 15:41:55 MDT 10.33.128.38(4461) db user 7438 startup> FATAL:  out of memory
<2009-07-31 15:41:55 MDT 10.33.128.38(4462) db user 7439 startup> FATAL:  out of memory
<2009-07-31 15:41:55 MDT 10.33.128.38(4463) db user 7440 startup> FATAL:  out of memory
<2009-07-31 15:41:55 MDT 10.20.219.194(3594) db user 7433 startup> FATAL:  out of memory
<2009-07-31 15:41:55 MDT 10.33.128.38(4464) db user 7441 startup> FATAL:  out of memory
<2009-07-31 15:41:55 MDT 10.33.128.38(4465) db user 7442 startup> FATAL:  out of memory
<2009-07-31 15:41:55 MDT 10.33.128.31(1263) db user 7447 startup> FATAL:  out of memory
<2009-07-31 15:41:55 MDT 10.44.5.43(3498) db user 7450 startup> FATAL:  out of memory
<2009-07-31 15:41:55 MDT 10.27.36.219(3860) db user 7448 startup> FATAL:  out of memory
<2009-07-31 15:41:55 MDT 10.33.128.10(35976) db user 7460 idle> LOG:  unexpected EOF on client connection
<2009-07-31 15:41:55 MDT    7253 > LOG:  server process (PID 4737) was terminated by signal 11
<2009-07-31 15:41:55 MDT    7253 > LOG:  terminating any other active server processes
<2009-07-31 15:41:55 MDT 10.33.128.10(35976) db user 7460 idle> WARNING:  terminating connection because of crash of another server process


Before crash:

date & free -m
Fri Jul 31 15:40:01 MDT 2009
 15:40:01 up  4:48,  3 users,  load average: 2.64, 3.04, 3.58

             total       used       free     shared    buffers     cached
Mem:         32187      22292       9895          0        164      19824
-/+ buffers/cache:       2303      29884
Swap:         1983          0       1983

After crash:

date & free -m
Fri Jul 31 15:45:01 MDT 2009
 15:45:01 up  4:53,  3 users,  load average: 4.45, 3.99, 3.80

             total       used       free     shared    buffers     cached
Mem:         32187      14726      17460          0        165      13850
-/+ buffers/cache:        710      31477
Swap:         1983          0       1983

thanks in advance
greetings









From: fabrixio1@hotmail.com
To: alvherre@commandprompt.com
CC: pgsql-admin@postgresql.org
Subject: Re: [ADMIN] out of memory
Date: Fri, 20 Mar 2009 09:15:05 -0700

.ExternalClass .EC_hmmessage P {padding:0px;} .ExternalClass body.EC_hmmessage {font-size:10pt;font-family:Verdana;}
Thanks for answer Alvaro
 
Already i have memory overcommit disabled.
I've always had with the value 2

vm.overcommit_memory = 2

Fabricio

> Date: Wed, 18 Mar 2009 20:28:02 -0400
> From: alvherre@commandprompt.com
> To: fabrixio1@hotmail.com
> CC: pgsql-admin@postgresql.org
> Subject: Re: [ADMIN] out of memory
>
> Fabricio wrote:
> >
> >
> > Hi
> >
> > I have a dedicated database server with 16 GB of RAM.
> >
> > and the oom-killer is killing my database
>
> Try disabling memory overcommit; see 17.4.3 here:
> http://www.postgresql.org/docs/8.3/interactive/kernel-resources.html
>
> --
> Alvaro Herrera http://www.CommandPrompt.com/
> The PostgreSQL Company - Command Prompt, Inc.



Date: Thu, 19 Mar 2009 21:36:00 +0300
Subject: Re: [ADMIN] out of memory
From: vladimir@greenmice.info
To: fabrixio1@hotmail.com
CC: pgsql-admin@postgresql.org



On Fri, Mar 13, 2009 at 11:15 PM, Fabricio <fabrixio1@hotmail.com> wrote:
 
Hello...
 
Linux 2.6.27.6 #7 SMP Sun Nov 16 00:48:35 MST 2008 i686 i686 i386 GNU/Linux
Slackware 11 and Postgres 8.1.15


Go download 64-bit OS.

--
Vladimir Rusinov
http://greenmice.info/


Messenger cumple 10 años de ser parte de tu vida

Re: out of memory

From
Vladimir Rusinov
Date:


On Sat, Aug 1, 2009 at 1:20 AM, Fabricio <fabrixio1@hotmail.com> wrote:


Hi

Some one know why this is happening?

I change the OS to 64 bits and now the oom-killer not hapend but Postgres is still showing out of memory

Linux SERVER 2.6.18-92.el5 #1 SMP Tue Apr 29 13:16:15 EDT 2008 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 5.2 (Tikanga)
PostgreSQL 8.1.15
32GB RAM

My postgresql.conf:
# - Memory -

shared_buffers = 1048576                        # min 16 or max_connections*2, 8KB each
temp_buffers = 1024                     # min 100, 8KB each
max_prepared_transactions = 20         # can be 0 or more
# note: increasing max_prepared_transactions costs ~600 bytes of shared memory
# per transaction slot, plus lock space (see max_locks_per_transaction).
work_mem = 1024                 # min 64, size in KB
maintenance_work_mem = 65536            # min 1024, size in KB
max_stack_depth = 2048                  # min 100, size in KB



kernel messages:
Jul 31 11:50:08 SERVER kernel: postmaster[7686]: segfault at 00007fff3feb1bb0 rip 00002b2f7e17e1a8 rsp 00007fff3feb1b90 error 6
Jul 31 15:41:55 SERVER kernel: postmaster[4737]: segfault at 00007fff3feb1bb0 rip 00002b2f7e1851a8 rsp 00007fff3feb1b90 error 6

PostgreSQL log:
<2009-07-31 15:41:55 MDT    7253 > LOG:  could not fork new process for connection: Cannot allocate memory
<2009-07-31 15:41:55 MDT    7253 > LOG:  could not fork new process for connection: Cannot allocate memory
<2009-07-31 15:41:55 MDT 10.27.41.74(2606) aforeglobal sysaforeglobal 7423 startup> FATAL:  out of memory
<2009-07-31 15:41:55 MDT    7253 > LOG:  could not fork new process for connection: Cannot allocate memory
<2009-07-31 15:41:55 MDT    7253 > LOG:  could not fork new process for connection: Cannot allocate memory
<2009-07-31 15:41:55 MDT 10.27.36.219(3859) db user 7424 startup> FATAL:  out of memory
<2009-07-31 15:41:55 MDT 10.20.216.82(1966) db user 7431 startup> FATAL:  out of memory
TopMemoryContext: 164432 total in 6 blocks; 5368 free (1 chunks); 159064 used
MdSmgr: 0 total in 0 blocks; 0 free (0 chunks); 0 used
LockTable (locallock hash): 8192 total in 1 blocks; 3744 free (0 chunks); 4448 used
Timezones: 52560 total in 2 blocks; 3744 free (0 chunks); 48816 used
ErrorContext: 8192 total in 1 blocks; 8160 free (4 chunks); 32 used
<2009-07-31 15:41:55 MDT 10.33.128.38(4458) db user 7434 startup> FATAL:  out of memory
<2009-07-31 15:41:55 MDT 10.33.128.38(4458) db user 7434 startup> DETAIL:  Failed on request of size 24000.
TopMemoryContext: 164432 total in 6 blocks; 5368 free (1 chunks); 159064 used
MdSmgr: 0 total in 0 blocks; 0 free (0 chunks); 0 used
LockTable (locallock hash): 8192 total in 1 blocks; 3744 free (0 chunks); 4448 used
Timezones: 52560 total in 2 blocks; 3744 free (0 chunks); 48816 used
ErrorContext: 8192 total in 1 blocks; 8160 free (4 chunks); 32 used
<2009-07-31 15:41:55 MDT 10.33.128.38(4459) db user 7435 startup> FATAL:  out of memory
<2009-07-31 15:41:55 MDT 10.33.128.38(4459) db user 7435 startup> DETAIL:  Failed on request of size 24000.
TopMemoryContext: 164432 total in 6 blocks; 5368 free (1 chunks); 159064 used
MdSmgr: 0 total in 0 blocks; 0 free (0 chunks); 0 used
LockTable (locallock hash): 8192 total in 1 blocks; 3744 free (0 chunks); 4448 used
Timezones: 52560 total in 2 blocks; 3744 free (0 chunks); 48816 used
ErrorContext: 8192 total in 1 blocks; 8160 free (4 chunks); 32 used
<2009-07-31 15:41:55 MDT 10.33.128.38(4460) db user 7436 startup> FATAL:  out of memory
<2009-07-31 15:41:55 MDT 10.33.128.38(4460) db user 7436 startup> DETAIL:  Failed on request of size 24000.
<2009-07-31 15:41:55 MDT 10.33.128.38(4461) db user 7438 startup> FATAL:  out of memory
<2009-07-31 15:41:55 MDT 10.33.128.38(4462) db user 7439 startup> FATAL:  out of memory
<2009-07-31 15:41:55 MDT 10.33.128.38(4463) db user 7440 startup> FATAL:  out of memory
<2009-07-31 15:41:55 MDT 10.20.219.194(3594) db user 7433 startup> FATAL:  out of memory
<2009-07-31 15:41:55 MDT 10.33.128.38(4464) db user 7441 startup> FATAL:  out of memory
<2009-07-31 15:41:55 MDT 10.33.128.38(4465) db user 7442 startup> FATAL:  out of memory
<2009-07-31 15:41:55 MDT 10.33.128.31(1263) db user 7447 startup> FATAL:  out of memory
<2009-07-31 15:41:55 MDT 10.44.5.43(3498) db user 7450 startup> FATAL:  out of memory
<2009-07-31 15:41:55 MDT 10.27.36.219(3860) db user 7448 startup> FATAL:  out of memory
<2009-07-31 15:41:55 MDT 10.33.128.10(35976) db user 7460 idle> LOG:  unexpected EOF on client connection
<2009-07-31 15:41:55 MDT    7253 > LOG:  server process (PID 4737) was terminated by signal 11
<2009-07-31 15:41:55 MDT    7253 > LOG:  terminating any other active server processes
<2009-07-31 15:41:55 MDT 10.33.128.10(35976) db user 7460 idle> WARNING:  terminating connection because of crash of another server process


Before crash:

date & free -m
Fri Jul 31 15:40:01 MDT 2009
 15:40:01 up  4:48,  3 users,  load average: 2.64, 3.04, 3.58

             total       used       free     shared    buffers     cached
Mem:         32187      22292       9895          0        164      19824
-/+ buffers/cache:       2303      29884
Swap:         1983          0       1983

After crash:

date & free -m
Fri Jul 31 15:45:01 MDT 2009
 15:45:01 up  4:53,  3 users,  load average: 4.45, 3.99, 3.80

             total       used       free     shared    buffers     cached
Mem:         32187      14726      17460          0        165      13850
-/+ buffers/cache:        710      31477
Swap:         1983          0       1983

Hm, It looks weird.
BTW, I assume that 24000 in "DETAIL:  Failed on request of size 24000" is size in bytes, but may be it's size in kilobytes? o_O

How many connections (active and max) do you have?
Are you sure that nothing else could eat memory (e.g. some poorly-written cronjob?)
How many VIRT memory does postmaster have before the crash?

Here is some suggersions:
Check memory limits in /etc/security/limits.conf and in `ulimit -a`.
tail /var/log/audit/audit.log (it might be something interesting there especially if you are running selinux)
Try to upgrade to 8.1.17 (it should be safe and fast operation).
Check your kernerl's shm settings.
Check fsm settings.

HTH


--
Vladimir Rusinov
http://greenmice.info/

Re: out of memory

From
Emanuel Calvo Franco
Date:
>
> Some one know why this is happening?
>
> I change the OS to 64 bits and now the oom-killer not hapend but Postgres is
> still showing out of memory
>
> Linux SERVER 2.6.18-92.el5 #1 SMP Tue Apr 29 13:16:15 EDT 2008 x86_64 x86_64
> x86_64 GNU/Linux
> Red Hat Enterprise Linux Server release 5.2 (Tikanga)
> PostgreSQL 8.1.15
> 32GB RAM
>
> My postgresql.conf:
> # - Memory -
>
> shared_buffers = 1048576                        # min 16 or
> max_connections*2, 8KB each
> temp_buffers = 1024                     # min 100, 8KB each
> max_prepared_transactions = 20         # can be 0 or more
> # note: increasing max_prepared_transactions costs ~600 bytes of shared
> memory
> # per transaction slot, plus lock space (see max_locks_per_transaction).
> work_mem = 1024                 # min 64, size in KB
> maintenance_work_mem = 65536            # min 1024, size in KB
> max_stack_depth = 2048                  # min 100, size in KB
>

What about the shmmax and shmall kernel parametters?
Which values you are managing?

--
              Emanuel Calvo Franco
      www.emanuelcalvofranco.com.ar

Re: out of memory

From
Fabricio
Date:


thanks Vladimir and Emanuel

    >How many connections (active and max) do you have?
max_connections = 2000
Active 1000

  >Are you sure that nothing else could eat memory (e.g. some poorly-written cronjob?)
There are 2 or 3  ligth cronjob's
but the free command show free memory... why? :s


  >How many VIRT memory does postmaster have before the crash?
8518m

 >Here is some suggersions:
 >Check memory limits in /etc/security/limits.conf and in `ulimit -a`.
The config file is the default
[postgres@SERVER ~]$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 268287
max locked memory       (kbytes, -l) 32
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 268287
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited


 >tail /var/log/audit/audit.log (it might be something interesting there especially if you are running selinux)
selinux is disabled

 >Try to upgrade to 8.1.17 (it should be safe and fast operation).
I will update

  >Check your kernerl's shm settings.
kernel.shmmax = 68719476736
kernel.shmall = 4294967296
Are the default, i think are too big already

 >Check fsm settings.
max_fsm_pages = 180000



regards...





Date: Sat, 1 Aug 2009 12:20:27 +0400
Subject: Re: [ADMIN] out of memory
From: vladimir@greenmice.info
To: fabrixio1@hotmail.com
CC: pgsql-admin@postgresql.org; alvherre@commandprompt.com



On Sat, Aug 1, 2009 at 1:20 AM, Fabricio <fabrixio1@hotmail.com> wrote:


Hi

Some one know why this is happening?

I change the OS to 64 bits and now the oom-killer not hapend but Postgres is still showing out of memory

Linux SERVER 2.6.18-92.el5 #1 SMP Tue Apr 29 13:16:15 EDT 2008 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 5.2 (Tikanga)
PostgreSQL 8.1.15
32GB RAM

My postgresql.conf:
# - Memory -

shared_buffers = 1048576                        # min 16 or max_connections*2, 8KB each
temp_buffers = 1024                     # min 100, 8KB each
max_prepared_transactions = 20         # can be 0 or more
# note: increasing max_prepared_transactions costs ~600 bytes of shared memory
# per transaction slot, plus lock space (see max_locks_per_transaction).
work_mem = 1024                 # min 64, size in KB
maintenance_work_mem = 65536            # min 1024, size in KB
max_stack_depth = 2048                  # min 100, size in KB



kernel messages:
Jul 31 11:50:08 SERVER kernel: postmaster[7686]: segfault at 00007fff3feb1bb0 rip 00002b2f7e17e1a8 rsp 00007fff3feb1b90 error 6
Jul 31 15:41:55 SERVER kernel: postmaster[4737]: segfault at 00007fff3feb1bb0 rip 00002b2f7e1851a8 rsp 00007fff3feb1b90 error 6

PostgreSQL log:
<2009-07-31 15:41:55 MDT    7253 > LOG:  could not fork new process for connection: Cannot allocate memory
<2009-07-31 15:41:55 MDT    7253 > LOG:  could not fork new process for connection: Cannot allocate memory
<2009-07-31 15:41:55 MDT 10.27.41.74(2606) aforeglobal sysaforeglobal 7423 startup> FATAL:  out of memory
<2009-07-31 15:41:55 MDT    7253 > LOG:  could not fork new process for connection: Cannot allocate memory
<2009-07-31 15:41:55 MDT    7253 > LOG:  could not fork new process for connection: Cannot allocate memory
<2009-07-31 15:41:55 MDT 10.27.36.219(3859) db user 7424 startup> FATAL:  out of memory
<2009-07-31 15:41:55 MDT 10.20.216.82(1966) db user 7431 startup> FATAL:  out of memory
TopMemoryContext: 164432 total in 6 blocks; 5368 free (1 chunks); 159064 used
MdSmgr: 0 total in 0 blocks; 0 free (0 chunks); 0 used
LockTable (locallock hash): 8192 total in 1 blocks; 3744 free (0 chunks); 4448 used
Timezones: 52560 total in 2 blocks; 3744 free (0 chunks); 48816 used
ErrorContext: 8192 total in 1 blocks; 8160 free (4 chunks); 32 used
<2009-07-31 15:41:55 MDT 10.33.128.38(4458) db user 7434 startup> FATAL:  out of memory
<2009-07-31 15:41:55 MDT 10.33.128.38(4458) db user 7434 startup> DETAIL:  Failed on request of size 24000.
TopMemoryContext: 164432 total in 6 blocks; 5368 free (1 chunks); 159064 used
MdSmgr: 0 total in 0 blocks; 0 free (0 chunks); 0 used
LockTable (locallock hash): 8192 total in 1 blocks; 3744 free (0 chunks); 4448 used
Timezones: 52560 total in 2 blocks; 3744 free (0 chunks); 48816 used
ErrorContext: 8192 total in 1 blocks; 8160 free (4 chunks); 32 used
<2009-07-31 15:41:55 MDT 10.33.128.38(4459) db user 7435 startup> FATAL:  out of memory
<2009-07-31 15:41:55 MDT 10.33.128.38(4459) db user 7435 startup> DETAIL:  Failed on request of size 24000.
TopMemoryContext: 164432 total in 6 blocks; 5368 free (1 chunks); 159064 used
MdSmgr: 0 total in 0 blocks; 0 free (0 chunks); 0 used
LockTable (locallock hash): 8192 total in 1 blocks; 3744 free (0 chunks); 4448 used
Timezones: 52560 total in 2 blocks; 3744 free (0 chunks); 48816 used
ErrorContext: 8192 total in 1 blocks; 8160 free (4 chunks); 32 used
<2009-07-31 15:41:55 MDT 10.33.128.38(4460) db user 7436 startup> FATAL:  out of memory
<2009-07-31 15:41:55 MDT 10.33.128.38(4460) db user 7436 startup> DETAIL:  Failed on request of size 24000.
<2009-07-31 15:41:55 MDT 10.33.128.38(4461) db user 7438 startup> FATAL:  out of memory
<2009-07-31 15:41:55 MDT 10.33.128.38(4462) db user 7439 startup> FATAL:  out of memory
<2009-07-31 15:41:55 MDT 10.33.128.38(4463) db user 7440 startup> FATAL:  out of memory
<2009-07-31 15:41:55 MDT 10.20.219.194(3594) db user 7433 startup> FATAL:  out of memory
<2009-07-31 15:41:55 MDT 10.33.128.38(4464) db user 7441 startup> FATAL:  out of memory
<2009-07-31 15:41:55 MDT 10.33.128.38(4465) db user 7442 startup> FATAL:  out of memory
<2009-07-31 15:41:55 MDT 10.33.128.31(1263) db user 7447 startup> FATAL:  out of memory
<2009-07-31 15:41:55 MDT 10.44.5.43(3498) db user 7450 startup> FATAL:  out of memory
<2009-07-31 15:41:55 MDT 10.27.36.219(3860) db user 7448 startup> FATAL:  out of memory
<2009-07-31 15:41:55 MDT 10.33.128.10(35976) db user 7460 idle> LOG:  unexpected EOF on client connection
<2009-07-31 15:41:55 MDT    7253 > LOG:  server process (PID 4737) was terminated by signal 11
<2009-07-31 15:41:55 MDT    7253 > LOG:  terminating any other active server processes
<2009-07-31 15:41:55 MDT 10.33.128.10(35976) db user 7460 idle> WARNING:  terminating connection because of crash of another server process


Before crash:

date & free -m
Fri Jul 31 15:40:01 MDT 2009
 15:40:01 up  4:48,  3 users,  load average: 2.64, 3.04, 3.58

             total       used       free     shared    buffers     cached
Mem:         32187      22292       9895          0        164      19824
-/+ buffers/cache:       2303      29884
Swap:         1983          0       1983

After crash:

date & free -m
Fri Jul 31 15:45:01 MDT 2009
 15:45:01 up  4:53,  3 users,  load average: 4.45, 3.99, 3.80

             total       used       free     shared    buffers     cached
Mem:         32187      14726      17460          0        165      13850
-/+ buffers/cache:        710      31477
Swap:         1983          0       1983

Hm, It looks weird.
BTW, I assume that 24000 in "DETAIL:  Failed on request of size 24000" is size in bytes, but may be it's size in kilobytes? o_O

How many connections (active and max) do you have?
Are you sure that nothing else could eat memory (e.g. some poorly-written cronjob?)
How many VIRT memory does postmaster have before the crash?

Here is some suggersions:
Check memory limits in /etc/security/limits.conf and in `ulimit -a`.
tail /var/log/audit/audit.log (it might be something interesting there especially if you are running selinux)
Try to upgrade to 8.1.17 (it should be safe and fast operation).
Check your kernerl's shm settings.
Check fsm settings.

HTH


--
Vladimir Rusinov
http://greenmice.info/


Únete a la celebración de Messenger y sigue siendo parte de esta historia