Postgres Pro
Downloads
Home   >   mailing lists

Thread: Unexpected data beyond EOF during heavy writes

Unexpected data beyond EOF during heavy writes

From
Tony Sullivan
Date:

Hello everyone,

 

We are seeing the following error message occasionally in the postgres logs:

 

2010-05-13 23:49:03 PDT ERROR: unexpected data beyond EOF in block 4106698 of relation "custom_discoveryprofile"

2010-05-13 23:49:03 PDT HINT: This has been seen to occur with buggy kernels; consider updating your system.

I have read several threads on this, including:
http://archives.postgresql.org/pgsql-general/2007-03/msg01535.php
and

http://archives.postgresql.org/pgsql-general/2009-07/msg01011.php

 

I am wondering if anyone has ever placed a bug report to any of the linux vendors or open source sites?  

I could not find a bug report that matched this at http://bugzilla.redhat.com/ and am trying to track down what is needed to resolve the error.


We are running a an IBM Blade Center server 64 bit Red Hat 5.3 kernel with 16 cores doing heavy I/O to the database.
uname -a
Linux 2.6.18-128.el5 #1 SMP Wed Dec 17 11:41:38 EST 2008 x86_64 x86_64 x86_64 GNU/Linux x86_64 x86_64 x86_64 GNU/Linux

Thanks for any pointers,

 

Tony Sullivan

Re: Unexpected data beyond EOF during heavy writes

From
Alvaro Herrera
Date:
Excerpts from Tony Sullivan's message of mar may 18 13:19:13 -0400 2010:
> Hello everyone,
> 
> We are seeing the following error message occasionally in the postgres logs:
> 
> 2010-05-13 23:49:03 PDT ERROR: unexpected data beyond EOF in block 4106698 of relation "custom_discoveryprofile"
> 2010-05-13 23:49:03 PDT HINT: This has been seen to occur with buggy kernels; consider updating your system.

What's your storage?

--

Re: Unexpected data beyond EOF during heavy writes

From
Tony Sullivan
Date:
<div class="Section1"><pre><font face="Courier New" size="2"><span style="font-size:10.0pt">> Hello
everyone,</span></font></pre><pre><fontface="Courier New" size="2"><span style="font-size:10.0pt">>
</span></font></pre><pre><fontface="Courier New" size="2"><span style="font-size:10.0pt">> We are seeing the
followingerror message occasionally in the postgres logs:</span></font></pre><pre><font face="Courier New"
size="2"><spanstyle="font-size:10.0pt">> </span></font></pre><pre><font face="Courier New" size="2"><span
style="font-size:10.0pt">>2010-05-13 23:49:03 PDT ERROR: unexpected data beyond EOF in block 4106698 of relation
"custom_discoveryprofile"</span></font></pre><pre><fontface="Courier New" size="2"><span style="font-size:10.0pt">>
2010-05-1323:49:03 PDT HINT: This has been seen to occur with buggy kernels; consider updating your
system.</span></font></pre><pre><fontface="Courier New" size="2"><span
style="font-size:10.0pt"> </span></font></pre><pre><fontface="Courier New" size="2"><span
style="font-size:10.0pt">What'syour storage?</span></font></pre><pre><font face="Courier New" size="2"><span
style="font-size:10.0pt"> </span></font></pre><pre><fontface="Courier New" size="2"><span style="font-size:10.0pt">--
</span></font></pre><pre><fontface="Courier New" size="2"><span style="font-size:10.0pt"> </span></font></pre><p
class="MsoNormal"style="text-autospace:none"><font face="Arial" size="2"><span
style="font-size:10.0pt;font-family:Arial">Itis NetApp storage - </span></font><font face="Courier New" size="2"><span
style="font-size:10.0pt;font-family:"CourierNew"">a FAS3070 running Data ONTAP 7.3.2</span></font><p class="MsoNormal"
style="text-autospace:none"><fontface="Courier New" size="2"><span style="font-size:10.0pt;font-family:"Courier
New""> </span></font><pclass="MsoNormal" style="text-autospace:none"><font face="Courier New" size="2"><span
style="font-size:10.0pt;font-family:"CourierNew"">Here are the mount options</span></font><p class="MsoNormal"
style="text-autospace:none"><fontface="Courier New" size="2"><span style="font-size:10.0pt;font-family:"Courier
New""> </span></font><pclass="MsoNormal" style="text-autospace:none"><font face="Courier New" size="2"><span
style="font-size:10.0pt;font-family:"CourierNew""><server>:/vol/sw on /x/eng/sw type nfs</span></font><p
class="MsoNormal"style="text-autospace:none"><font face="Courier New" size="2"><span
style="font-size:10.0pt;font-family:"Courier
New"">(rw,intr,hard,rsize=32768,wsize=32768,nfsvers=3,timeo=600,tcp,nointr,addr=xx.xx.</span></font><pclass="MsoNormal"
style="text-autospace:none"><fontface="Courier New" size="2"><span style="font-size:10.0pt;font-family:"Courier
New"">xx.xx)</span></font><pclass="MsoNormal" style="text-autospace:none"><font face="Courier New" size="2"><span
style="font-size:10.0pt;font-family:"CourierNew""> </span></font><p class="MsoNormal"><font face="Arial" size="2"><span
style="font-size:10.0pt;
font-family:Arial">Thanks,</span></font><p class="MsoNormal"><font face="Arial" size="2"><span style="font-size:10.0pt;
font-family:Arial"> </span></font><p class="MsoNormal"><strong><b><font face="Arial" size="2"><span
style="font-size:10.0pt;font-family:Arial">TonySullivan</span></font></b></strong> <strong><b><font face="Arial"
size="2"><spanstyle="font-size:10.0pt;font-family: 
Arial"></span></font></b></strong><p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size:
12.0pt"> </span></font></div>

Re: Unexpected data beyond EOF during heavy writes

From
Alvaro Herrera
Date:
Excerpts from Tony Sullivan's message of jue may 20 16:54:17 -0400 2010:
> > Hello everyone,
> 
> >
> 
> > We are seeing the following error message occasionally in the postgres logs:
> 
> >
> 
> > 2010-05-13 23:49:03 PDT ERROR: unexpected data beyond EOF in block 4106698 of relation "custom_discoveryprofile"
> 
> > 2010-05-13 23:49:03 PDT HINT: This has been seen to occur with buggy kernels; consider updating your system.
> 
> What's your storage?

This was added here
http://archives.postgresql.org/message-id/20060925220110.76B6A9FB251@postgresql.org
in response to these two:
http://thread.gmane.org/gmane.comp.db.postgresql.admin/18807
http://thread.gmane.org/gmane.comp.db.postgresql.general/74532

We (at Command Prompt) researched this recently for another setup and
the common point you both have is NetApp.  I then wondered about a bug
in NetApp driver or NFS client implementation.

--

Re: Unexpected data beyond EOF during heavy writes

From
Rosser Schwarz
Date:
On Thu, May 20, 2010 at 3:19 PM, Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:

> We (at Command Prompt) researched this recently for another setup and
> the common point you both have is NetApp.  I then wondered about a bug
> in NetApp driver or NFS client implementation.

It's definitely not (just) NetApp, though it may be their NFS -- or
NFS in general; I couldn't say.  I can't speak to their NFS
implementation, beyond having generally heard good things about it,
but I've run PostgreSQL on filers for years, and have never seen that
message.  Granted, I've only been iSCSI- or fibre-attached (or had the
storage path abstracted away by some form of virtualization), so I
haven't seen every possible use-case.

In general, though, I'd be pretty wary of running postgres on an NFS
mount.  I know a lot of people run Oracle that way, but at the
filesystem level, there are some vast differences between the two.

Has anyone ever seen this message on non-NetApp NFS?

rls

--
:wq

Re: Unexpected data beyond EOF during heavy writes

From
Tom Lane
Date:
Rosser Schwarz <rosser.schwarz@gmail.com> writes:
> Has anyone ever seen this message on non-NetApp NFS?

It's been seen on non-NFS storage:
http://archives.postgresql.org/pgsql-admin/2006-09/msg00096.php

I don't believe we implicated NFS in the other original report,
either.  However, it's certainly possible that there's a similar
bug in the NFS stack too on some platforms.
        regards, tom lane