Thread: fdatasync failed, I/O error

fdatasync failed, I/O error

From
"Brusser, Michael"
Date:
<br /><p><font size="2">Our customer is running Postgresql v. 7.3.4 on Linux(Redhat 3.0 Ent.)</font><br /><font
size="2">Theapplication crashed and I am trying to understand what could have caused this kind of
problems:</font><p><fontsize="2">   fdatasync of log file 0, segment 2 failed: input/output error</font><br /><font
size="2">  PGSTAT: Error closing temp stats file </font><br /><font size="2">     
.../.../postgresql/global/pgstat.tmp.10278:Input/output error</font><p><font size="2">I suppose they could be using NFS
(Iknow, I know... we have to deal with that. But could it be something else? A bad disk?)</font><p><font size="2">Thank
you,</font><br/><font size="2">Mike</font><p><font
size="2">======================================================================</font><p><fontsize="2">From the
databaselog:</font><br /><font size="2">-------------------------</font><br /><font size="2">... ... ... ...</font><br
/><fontsize="2">2005-06-01 11:55:25 PANIC:  fdatasync of log file 0, segment 2 failed: input/output error</font><br
/><fontsize="2">2005-06-01 11:55:25 LOG:  server process (pid 16178) was terminated by signal 6</font><br /><font
size="2">2005-06-0111:55:25 WARNING:  Message from PostgreSQL backend:</font><br /><font size="2">        The
Postmasterhas informed me that some other backend</font><br /><font size="2">        died abnormally and possibly
corruptedshared memory.</font><br /><font size="2">        I have rolled back the current transaction and am</font><br
/><fontsize="2">        going to terminate your database system connection and exit.</font><br /><font size="2">       
Pleasereconnect to the database system and repeat your query.</font><br /><font size="2">2005-06-01 11:55:25 WARNING: 
Messagefrom PostgreSQL backend:</font><br /><font size="2">        The Postmaster has informed me that some other
backend</font><br/><font size="2">        died abnormally and possibly corrupted shared memory.</font><br /><font
size="2">       I have rolled back the current transaction and am</font><br /><font size="2">        going to terminate
yourdatabase system connection and exit.</font><br /><font size="2">        Please reconnect to the database system and
repeatyour query.</font><br /><font size="2">2005-06-01 11:55:25 WARNING:  Message from PostgreSQL backend:</font><br
/><fontsize="2">        The Postmaster has informed me that some other backend</font><br /><font size="2">        died
abnormallyand possibly corrupted shared memory.</font><br /><font size="2">        I have rolled back the current
transactionand am</font><br /><font size="2">        going to terminate your database system connection and
exit.</font><br/><font size="2">        Please reconnect to the database system and repeat your query.</font><br
/><fontsize="2">2005-06-01 11:55:25 WARNING:  Message from PostgreSQL backend:</font><br /><font size="2">        The
Postmasterhas informed me that some other backend</font><br /><font size="2">        died abnormally and possibly
corruptedshared memory.</font><br /><font size="2">        I have rolled back the current transaction and am</font><br
/><fontsize="2">        going to terminate your database system connection and exit.</font><br /><font size="2">       
Pleasereconnect to the database system and repeat your query.</font><br /><font size="2">2005-06-01 11:55:25 WARNING: 
Messagefrom PostgreSQL backend:</font><br /><font size="2">        The Postmaster has informed me that some other
backend</font><br/><font size="2">        died abnormally and possibly corrupted shared memory.</font><br /><font
size="2">       I have rolled back the current transaction and am</font><br /><font size="2">        going to terminate
yourdatabase system connection and exit.</font><br /><font size="2">        Please reconnect to the database system and
repeatyour query.</font><br /><font size="2">2005-06-01 11:55:28 LOG:  all server processes terminated; reinitializing
sharedmemory and semaphores</font><br /><font size="2">2005-06-01 11:55:28 LOG:  database system was interrupted at
2005-06-0102:05:21 IDT</font><br /><font size="2">2005-06-01 11:55:28 LOG:  checkpoint record is at 0/268BF24</font><br
/><fontsize="2">2005-06-01 11:55:28 LOG:  redo record is at 0/268BF24; undo record is at 0/0; shutdown FALSE</font><br
/><fontsize="2">2005-06-01 11:55:28 LOG:  next transaction id: 23827; next oid: 25973</font><br /><font
size="2">2005-06-0111:55:28 LOG:  database system was not properly shut down; automatic recovery in progress</font><br
/><fontsize="2">2005-06-01 11:55:28 LOG:  redo starts at 0/268BF64</font><br /><font size="2">2005-06-01 11:55:28 LOG: 
ReadRecord:record with zero length at 0/269A9EC</font><br /><font size="2">2005-06-01 11:55:28 LOG:  redo done at
0/269A9AC</font><br/><font size="2">2005-06-01 11:55:31 PANIC:  fdatasync of log file 0, segment 2 failed: Input/output
error</font><br/><font size="2">2005-06-01 11:55:31 LOG:  startup process (pid 16180) was terminated by signal
6</font><br/><font size="2">2005-06-01 11:55:32 LOG:  PGSTAT: Error closing temp stats file
/apps/sync/syncdata/sync/10000/server_metadata/postgr</font><br/><font size="2">esql/global/pgstat.tmp.10278:
Input/outputerror</font><br /><font size="2">2005-06-01 12:47:28 LOG:  database system shutdown was interrupted at
2005-06-0111:55:28 IDT</font><br /><font size="2">2005-06-01 12:47:28 LOG:  checkpoint record is at 0/268BF24</font><br
/><fontsize="2">2005-06-01 12:47:28 LOG:  redo record is at 0/268BF24; undo record is at 0/0; shutdown FALSE</font><br
/><fontsize="2">2005-06-01 12:47:28 LOG:  next transaction id: 23827; next oid: 25973</font><br /><font
size="2">2005-06-0112:47:28 LOG:  database system was not properly shut down; automatic recovery in progress</font><br
/><fontsize="2">2005-06-01 12:47:28 LOG:  redo starts at 0/268BF64</font><br /><font size="2">2005-06-01 12:47:28 LOG: 
ReadRecord:record with zero length at 0/269AA2C</font><br /><font size="2">2005-06-01 12:47:28 LOG:  redo done at
0/269A9EC</font><br/><font size="2">2005-06-01 12:47:30 LOG:  database system is ready</font> 

Re: fdatasync failed, I/O error

From
Tom Lane
Date:
"Brusser, Michael" <Michael.Brusser@matrixone.com> writes:
> Our customer is running Postgresql v. 7.3.4 on Linux(Redhat 3.0 Ent.)
> The application crashed and I am trying to understand what could have caused
> this kind of problems:

>    fdatasync of log file 0, segment 2 failed: input/output error
>    PGSTAT: Error closing temp stats file 
>       .../.../postgresql/global/pgstat.tmp.10278: Input/output error

"input/output error" definitely means a hardware problem ... you might
try looking in the kernel logs (/var/log/messages) to see if any
additional information got recorded there.
        regards, tom lane


Re: fdatasync failed, I/O error

From
Alvaro Herrera
Date:
On Wed, Jun 01, 2005 at 10:27:04AM -0400, Brusser, Michael wrote:
> 
> Our customer is running Postgresql v. 7.3.4 on Linux(Redhat 3.0 Ent.)
> The application crashed and I am trying to understand what could have caused
> this kind of problems:
> 
>    fdatasync of log file 0, segment 2 failed: input/output error
>    PGSTAT: Error closing temp stats file 
>       .../.../postgresql/global/pgstat.tmp.10278: Input/output error
> 
> I suppose they could be using NFS (I know, I know... we have to deal with
> that. But could it be something else? A bad disk?)

Bad disk, bad controller, bad cable between the two, are some which come
to mind.

-- 
Alvaro Herrera (<alvherre[a]surnet.cl>)
"The Gord often wonders why people threaten never to come back after they've
been told never to return" (www.actsofgord.com)