Thread: out of memory

Hi

I have a dedicated database server with 16 GB of RAM.

and the oom-killer is killing my database

 
Hello...
 
Linux 2.6.27.6 #7 SMP Sun Nov 16 00:48:35 MST 2008 i686 i686 i386 GNU/Linux
Slackware 11 and Postgres 8.1.15

Hi

Some one know why this is happening?

I change the OS to 64 bits and now the oom-killer not hapend but Postgres is still showing out of memory

Linux SERVER 2.6.18-92.el5 #1 SMP Tue Apr 29 13:16:15 EDT 2008 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 5.2 (Tikanga)
PostgreSQL 8.1.15
32GB RAM

My postgresql.conf:
# - Memory -

shared_buffers = 1048576                        # min 16 or max_connections*2, 8KB each
temp_buffers = 1024                     # min 100, 8KB each
max_prepared_transactions = 20         # can be 0 or more
# note: increasing max_prepared_transactions costs ~600 bytes of shared memory
# per transaction slot, plus lock space (see max_locks_per_transaction).
work_mem = 1024                 # min 64, size in KB
maintenance_work_mem = 65536            # min 1024, size in KB
max_stack_depth = 2048                  # min 100, size in KB



kernel messages:
Jul 31 11:50:08 SERVER kernel: postmaster[7686]: segfault at 00007fff3feb1bb0 rip 00002b2f7e17e1a8 rsp 00007fff3feb1b90 error 6
Jul 31 15:41:55 SERVER kernel: postmaster[4737]: segfault at 00007fff3feb1bb0 rip 00002b2f7e1851a8 rsp 00007fff3feb1b90 error 6

PostgreSQL log:
<2009-07-31 15:41:55 MDT    7253 > LOG:  could not fork new process for connection: Cannot allocate memory
<2009-07-31 15:41:55 MDT    7253 > LOG:  could not fork new process for connection: Cannot allocate memory
<2009-07-31 15:41:55 MDT 10.27.41.74(2606) aforeglobal sysaforeglobal 7423 startup> FATAL:  out of memory
<2009-07-31 15:41:55 MDT    7253 > LOG:  could not fork new process for connection: Cannot allocate memory
<2009-07-31 15:41:55 MDT    7253 > LOG:  could not fork new process for connection: Cannot allocate memory
<2009-07-31 15:41:55 MDT 10.27.36.219(3859) db user 7424 startup> FATAL:  out of memory
<2009-07-31 15:41:55 MDT 10.20.216.82(1966) db user 7431 startup> FATAL:  out of memory
TopMemoryContext: 164432 total in 6 blocks; 5368 free (1 chunks); 159064 used
MdSmgr: 0 total in 0 blocks; 0 free (0 chunks); 0 used
LockTable (locallock hash): 8192 total in 1 blocks; 3744 free (0 chunks); 4448 used
Timezones: 52560 total in 2 blocks; 3744 free (0 chunks); 48816 used
ErrorContext: 8192 total in 1 blocks; 8160 free (4 chunks); 32 used
<2009-07-31 15:41:55 MDT 10.33.128.38(4458) db user 7434 startup> FATAL:  out of memory
<2009-07-31 15:41:55 MDT 10.33.128.38(4458) db user 7434 startup> DETAIL:  Failed on request of size 24000.
TopMemoryContext: 164432 total in 6 blocks; 5368 free (1 chunks); 159064 used
MdSmgr: 0 total in 0 blocks; 0 free (0 chunks); 0 used
LockTable (locallock hash): 8192 total in 1 blocks; 3744 free (0 chunks); 4448 used
Timezones: 52560 total in 2 blocks; 3744 free (0 chunks); 48816 used
ErrorContext: 8192 total in 1 blocks; 8160 free (4 chunks); 32 used
<2009-07-31 15:41:55 MDT 10.33.128.38(4459) db user 7435 startup> FATAL:  out of memory
<2009-07-31 15:41:55 MDT 10.33.128.38(4459) db user 7435 startup> DETAIL:  Failed on request of size 24000.
TopMemoryContext: 164432 total in 6 blocks; 5368 free (1 chunks); 159064 used
MdSmgr: 0 total in 0 blocks; 0 free (0 chunks); 0 used
LockTable (locallock hash): 8192 total in 1 blocks; 3744 free (0 chunks); 4448 used
Timezones: 52560 total in 2 blocks; 3744 free (0 chunks); 48816 used
ErrorContext: 8192 total in 1 blocks; 8160 free (4 chunks); 32 used
<2009-07-31 15:41:55 MDT 10.33.128.38(4460) db user 7436 startup> FATAL:  out of memory
<2009-07-31 15:41:55 MDT 10.33.128.38(4460) db user 7436 startup> DETAIL:  Failed on request of size 24000.
<2009-07-31 15:41:55 MDT 10.33.128.38(4461) db user 7438 startup> FATAL:  out of memory
<2009-07-31 15:41:55 MDT 10.33.128.38(4462) db user 7439 startup> FATAL:  out of memory
<2009-07-31 15:41:55 MDT 10.33.128.38(4463) db user 7440 startup> FATAL:  out of memory
<2009-07-31 15:41:55 MDT 10.20.219.194(3594) db user 7433 startup> FATAL:  out of memory
<2009-07-31 15:41:55 MDT 10.33.128.38(4464) db user 7441 startup> FATAL:  out of memory
<2009-07-31 15:41:55 MDT 10.33.128.38(4465) db user 7442 startup> FATAL:  out of memory
<2009-07-31 15:41:55 MDT 10.33.128.31(1263) db user 7447 startup> FATAL:  out of memory
<2009-07-31 15:41:55 MDT 10.44.5.43(3498) db user 7450 startup> FATAL:  out of memory
<2009-07-31 15:41:55 MDT 10.27.36.219(3860) db user 7448 startup> FATAL:  out of memory
<2009-07-31 15:41:55 MDT 10.33.128.10(35976) db user 7460 idle> LOG:  unexpected EOF on client connection
<2009-07-31 15:41:55 MDT    7253 > LOG:  server process (PID 4737) was terminated by signal 11
<2009-07-31 15:41:55 MDT    7253 > LOG:  terminating any other active server processes
<2009-07-31 15:41:55 MDT 10.33.128.10(35976) db user 7460 idle> WARNING:  terminating connection because of crash of another server process


Before crash:

date & free -m
Fri Jul 31 15:40:01 MDT 2009
 15:40:01 up  4:48,  3 users,  load average: 2.64, 3.04, 3.58

             total       used       free     shared    buffers     cached
Mem:         32187      22292       9895          0        164      19824
-/+ buffers/cache:       2303      29884
Swap:         1983          0       1983

After crash:

date & free -m
Fri Jul 31 15:45:01 MDT 2009
 15:45:01 up  4:53,  3 users,  load average: 4.45, 3.99, 3.80

             total       used       free     shared    buffers     cached
Mem:         32187      14726      17460          0        165      13850
-/+ buffers/cache:        710      31477
Swap:         1983          0       1983