Thread: PostgreSQL crashed server

Saturday our DB development server crashed (no big deal, it's not the
production server), and checking out the logs in /var/log/messages, I
get this:

Mar 24 02:34:48 bugs kernel: [297840.580276] postgres        D
c10201fe     0  1471   1463 0x00000000
Mar 24 02:34:48 bugs kernel: [297840.580281]  f723d960 00000086
c10201f2 c10201fe 00000000 00000000 c1091117 c144e2c0
Mar 24 02:34:48 bugs kernel: [297840.580289]  f723db00 343b8000
c13e6ba4 c144e2c0 f723d960 c144e2c0 c144e2c0 c13e7284
Mar 24 02:34:48 bugs kernel: [297840.580297]  00000000 00000000
c13e6ba4 ffffffff 00000002 00000000 00000040 00000010
Mar 24 02:34:48 bugs kernel: [297840.580305] Call Trace:
Mar 24 02:34:48 bugs kernel: [297840.580311]  [<c10201f2>] ?
__kunmap_atomic+0x5a/0x7b
Mar 24 02:34:48 bugs kernel: [297840.580315]  [<c10201fe>] ?
__kunmap_atomic+0x66/0x7b
Mar 24 02:34:48 bugs kernel: [297840.580319]  [<c1091117>] ?
get_page_from_freelist+0x31a/0x3ac
Mar 24 02:34:48 bugs kernel: [297840.580324]  [<c1044bfe>] ?
prepare_to_wait_exclusive+0x15/0x53
Mar 24 02:34:48 bugs kernel: [297840.580328]  [<c11e722d>] ?
__lock_sock+0x53/0x71
Mar 24 02:34:48 bugs kernel: [297840.580332]  [<c1044ade>] ?
autoremove_wake_function+0x0/0x29
Mar 24 02:34:48 bugs kernel: [297840.580336]  [<c11e729a>] ?
lock_sock_nested+0x1d/0x2d
Mar 24 02:34:48 bugs kernel: [297840.580340]  [<c12300ab>] ?
udp_sendmsg+0x8d/0x594
Mar 24 02:34:48 bugs kernel: [297840.580345]  [<c10d6dd5>] ?
__find_get_block_slow+0x105/0x113
Mar 24 02:34:48 bugs kernel: [297840.580349]  [<c10d71ad>] ?
__find_get_block+0x14d/0x157
Mar 24 02:34:48 bugs kernel: [297840.580353]  [<c12362f2>] ?
inet_sendmsg+0x6c/0x73
Mar 24 02:34:48 bugs kernel: [297840.580357]  [<c11e479f>] ?
__sock_sendmsg+0x48/0x4e
Mar 24 02:34:48 bugs kernel: [297840.580361]  [<c11e4ba4>] ?
sock_sendmsg+0x78/0x8f
Mar 24 02:34:48 bugs kernel: [297840.580372]  [<f84f4366>] ?
ext3_mark_inode_dirty+0x29/0x30 [ext3]
Mar 24 02:34:48 bugs kernel: [297840.580384]  [<f84fae83>] ?
__ext3_journal_stop+0x19/0x34 [ext3]
Mar 24 02:34:48 bugs kernel: [297840.580391]  [<c10d2a13>] ?
__mark_inode_dirty+0x1d/0x15f
Mar 24 02:34:48 bugs kernel: [297840.580395]  [<c11e4fd9>] ?
sockfd_lookup_light+0x13/0x3f
Mar 24 02:34:48 bugs kernel: [297840.580399]  [<c11e562a>] ?
sys_sendto+0xed/0x121
Mar 24 02:34:48 bugs kernel: [297840.580403]  [<c10938df>] ?
release_pages+0x104/0x10c
Mar 24 02:34:48 bugs kernel: [297840.580409]  [<c10ac189>] ?
free_pages_and_swap_cache+0x43/0x50
Mar 24 02:34:48 bugs kernel: [297840.580413]  [<c11e5677>] ? sys_send+0x19/0x1d
Mar 24 02:34:48 bugs kernel: [297840.580417]  [<c11e655d>] ?
sys_socketcall+0xee/0x1cb
Mar 24 02:34:48 bugs kernel: [297840.580421]  [<c10a6138>] ?
sys_munmap+0x3a/0x40
Mar 24 02:34:48 bugs kernel: [297840.580425]  [<c1002f1f>] ?
sysenter_do_call+0x12/0x28

Server information:

# uname -a
Linux bugs 2.6.38-1-686 #1 SMP Wed Mar 16 17:37:47 UTC 2011 i686 GNU/Linux
# cat /etc/debian_version
wheezy/sid
# dpkg -l | grep postgresql
ii  postgresql-9.1                        9.1.3-2
   object-relational SQL database, version 9.1 server


Nothing strange in postgresql.log. Last log is this (and there is was
no DB activity at that hour):

2012-03-24 02:31:26 ART  - : LOG:  checkpoint starting: time

Any ideas?

Last year I had memory problems with this server (changed the faulty
bank), I wouldn't want it to be happening again.

--
Martín Marqués
select 'martin.marques' || '@' || 'gmail.com'
DBA, Programador, Administrador

On 03/26/2012 07:41 PM, Martín Marqués wrote:
> Any ideas?
>
> Last year I had memory problems with this server (changed the faulty
> bank), I wouldn't want it to be happening again.

It's a crash deep in memory management for inode storage used by the
ext3 filesystem code. Chances are it's a hardware fault or (small
chance) kernel bug. If Pg could cause a crash like that it'd be a kernel
DoS vulnerability, but I'm much more inclined to suspect the hardware
even without the dodgy history.

--
Craig Ringer