Thread: xlog corruption

xlog corruption

From
Jameison Martin
Date:
I'd like to get some clarification around an architectural point about recovery. I see that it is normal to see "unexpected pageaddr" errors during recovery because of the way Postgres overwrites old log files, and thus this is taken to be a normal termination condition, i.e. the end of the log (see http://doxygen.postgresql.org/xlog_8c.html#a0519e464bfaa79bde3e241e6cff986c7). My question is how does recovery distinguish between the actual end of the log as opposed to a log file corruption (e.g. torn page)?  

I'd like to be able to distinguish between a corruption in the log vs. a normal recovery condition if possible.

Thanks.

Re: xlog corruption

From
Jeff Davis
Date:
On Mon, 2012-02-27 at 16:30 -0800, Jameison Martin wrote:
> I'd like to get some clarification around an architectural point about
> recovery. I see that it is normal to see "unexpected pageaddr" errors
> during recovery because of the way Postgres overwrites old log files,
> and thus this is taken to be a normal termination condition, i.e. the
> end of the log (see
> http://doxygen.postgresql.org/xlog_8c.html#a0519e464bfaa79bde3e241e6cff986c7). My question is how does recovery
distinguishbetween the actual end of the log as opposed to a log file corruption (e.g. torn page)?   
>
>
> I'd like to be able to distinguish between a corruption in the log vs.
> a normal recovery condition if possible.

If you have a power failure, a torn page in the WAL is expected. Torn
pages in the data pages are fixed up using WAL; but WAL doesn't have
anything under it to prevent/fix torn pages (unless your filesystem
prevents them).

Of course, checksums are used to prevent recovery from attempting to
play a partial or otherwise corrupt WAL record.

What kind of corruption are you trying to detect?

Regards,
    Jeff Davis



Re: xlog corruption

From
Jameison Martin
Date:

i'm not suggesting that Postgres should necessarily be able to repair a corruption in the WAL, my issue is that there is no way of knowing that there was a corruption in the WAL because the normal detection of the end of the WAL is reported the same way as a corruption in the middle of the WAL. 

i'd like some unequivocal indication that the WAL was corrupted.

thanks.

From: Jeff Davis <pgsql@j-davis.com>
To: Jameison Martin <jameisonb@yahoo.com>
Cc: "pgsql-general@postgresql.org" <pgsql-general@postgresql.org>
Sent: Wednesday, March 14, 2012 3:55 PM
Subject: Re: [GENERAL] xlog corruption

On Mon, 2012-02-27 at 16:30 -0800, Jameison Martin wrote:
> I'd like to get some clarification around an architectural point about
> recovery. I see that it is normal to see "unexpected pageaddr" errors
> during recovery because of the way Postgres overwrites old log files,
> and thus this is taken to be a normal termination condition, i.e. the
> end of the log (see
> http://doxygen.postgresql.org/xlog_8c.html#a0519e464bfaa79bde3e241e6cff986c7). My question is how does recovery distinguish between the actual end of the log as opposed to a log file corruption (e.g. torn page)? 
>
>
> I'd like to be able to distinguish between a corruption in the log vs.
> a normal recovery condition if possible.

If you have a power failure, a torn page in the WAL is expected. Torn
pages in the data pages are fixed up using WAL; but WAL doesn't have
anything under it to prevent/fix torn pages (unless your filesystem
prevents them).

Of course, checksums are used to prevent recovery from attempting to
play a partial or otherwise corrupt WAL record.

What kind of corruption are you trying to detect?

Regards,
    Jeff Davis