Thread: Error in log file after database crash

Error in log file after database crash

From
Rob Cowell
Date:
Hi,

One of my test databases "crashed" over the weekend after one of the sysadmin's in the team shut down the systemd
containerbefore stopping the database. 
My question is, how bad is this :)

Some background :
I have an application that uses Postgres as it's database.
Both the application, and postgres will be upgraded to newer versions in the coming weeks.
As part of a test-run I've upgraded the application to ensure that works, before I upgrade the database.
The application is undergoing some UAT testing prior to having the database upgraded.
Currently I'm running 9.1.10, and will be upgrading to the latest release of 9.3.x (not my choice of version, but the
applicationcurrently doesn't support 9.5.x) 

I've taken a full file-system backup of the database directories (with the database shut down), and a pg_dumpall.
Having restarted the database everything seems okay until the logs start showing these errors :


[2016-05-05 20:08:55 UTC]    LOCATION:  AbortBufferIO, bufmgr.c:2799
[2016-05-05 20:08:56 UTC]    ERROR:  XX000: xlog flush request 36D0/E8301358 is not satisfied --- flushed only to
36CA/94E12DC0
[2016-05-05 20:08:56 UTC]    CONTEXT:  writing block 0 of relation pg_tblspc/16395/PG_9.1_201105231/16400/33044487_vm
[2016-05-05 20:08:56 UTC]    LOCATION:  XLogFlush, xlog.c:2171
[2016-05-05 20:08:56 UTC]    WARNING:  58030: could not write block 0 of
pg_tblspc/16395/PG_9.1_201105231/16400/33044487_vm
[2016-05-05 20:08:56 UTC]    DETAIL:  Multiple failures --- write error might be permanent.
[2016-05-05 20:08:56 UTC]    LOCATION:  AbortBufferIO, bufmgr.c:2799
[2016-05-05 20:08:57 UTC]    ERROR:  XX000: xlog flush request 36D0/E8301358 is not satisfied --- flushed only to
36CA/94E12DC0
[2016-05-05 20:08:57 UTC]    CONTEXT:  writing block 0 of relation pg_tblspc/16395/PG_9.1_201105231/16400/33044487_vm
[2016-05-05 20:08:57 UTC]    LOCATION:  XLogFlush, xlog.c:2171
[2016-05-05 20:08:57 UTC]    WARNING:  58030: could not write block 0 of
pg_tblspc/16395/PG_9.1_201105231/16400/33044487_vm
[2016-05-05 20:08:57 UTC]    DETAIL:  Multiple failures --- write error might be permanent.
[2016-05-05 20:08:57 UTC]    LOCATION:  AbortBufferIO, bufmgr.c:2799
[2016-05-05 20:08:58 UTC]    ERROR:  XX000: xlog flush request 36D0/E8301358 is not satisfied --- flushed only to
36CA/94E12DC0
[2016-05-05 20:08:58 UTC]    CONTEXT:  writing block 0 of relation pg_tblspc/16395/PG_9.1_201105231/16400/33044487_vm
[2016-05-05 20:08:58 UTC]    LOCATION:  XLogFlush, xlog.c:2171
[2016-05-05 20:08:58 UTC]    WARNING:  58030: could not write block 0 of
pg_tblspc/16395/PG_9.1_201105231/16400/33044487_vm
[2016-05-05 20:08:58 UTC]    DETAIL:  Multiple failures --- write error might be permanent.
[2016-05-05 20:08:58 UTC]    LOCATION:  AbortBufferIO, bufmgr.c:2799
[2016-05-05 20:08:59 UTC]    ERROR:  XX000: xlog flush request 36D0/E8301358 is not satisfied --- flushed only to
36CA/94E12DC0
[2016-05-05 20:08:59 UTC]    CONTEXT:  writing block 0 of relation pg_tblspc/16395/PG_9.1_201105231/16400/33044487_vm
[2016-05-05 20:08:59 UTC]    LOCATION:  XLogFlush, xlog.c:2171
[2016-05-05 20:08:59 UTC]    WARNING:  58030: could not write block 0 of
pg_tblspc/16395/PG_9.1_201105231/16400/33044487_vm
[2016-05-05 20:08:59 UTC]    DETAIL:  Multiple failures --- write error might be permanent.
[2016-05-05 20:08:59 UTC]    LOCATION:  AbortBufferIO, bufmgr.c:2799
[2016-05-05 20:09:00 UTC]    ERROR:  XX000: xlog flush request 36D0/E8301358 is not satisfied --- flushed only to
36CA/94E12DC0
[2016-05-05 20:09:00 UTC]    CONTEXT:  writing block 0 of relation pg_tblspc/16395/PG_9.1_201105231/16400/33044487_vm
[2016-05-05 20:09:00 UTC]    LOCATION:  XLogFlush, xlog.c:2171
[2016-05-05 20:09:00 UTC]    WARNING:  58030: could not write block 0 of
pg_tblspc/16395/PG_9.1_201105231/16400/33044487_vm
[2016-05-05 20:09:00 UTC]    DETAIL:  Multiple failures --- write error might be permanent.
[2016-05-05 20:09:00 UTC]    LOCATION:  AbortBufferIO, bufmgr.c:2799

Cheers,
Rob.


Re: Error in log file after database crash

From
Michael Paquier
Date:
On Tue, May 10, 2016 at 1:11 AM, Rob Cowell <rob.cowell@transversal.com> wrote:
> Currently I'm running 9.1.10,

9.1.10 has been released in 2013. You are missing 2 years and a half
worth of various fixes, so you may want to update to 9.1.22 first.

> [2016-05-05 20:09:00 UTC]    LOCATION:  XLogFlush, xlog.c:2171
> [2016-05-05 20:09:00 UTC]    WARNING:  58030: could not write block 0 of
pg_tblspc/16395/PG_9.1_201105231/16400/33044487_vm
> [2016-05-05 20:09:00 UTC]    DETAIL:  Multiple failures --- write error might be permanent.
> [2016-05-05 20:09:00 UTC]    LOCATION:  AbortBufferIO, bufmgr.c:2799

This looks like corrupted data, but that's hard to tell with this
level of details. Rolling in a logical backup may be the best way to
go forward here.
--
Michael