Thread: xlog corruption
I'd like to get some clarification around an architectural point about recovery. I see that it is normal to see "unexpected pageaddr" errors during recovery because of the way Postgres overwrites old log files, and thus this is taken to be a normal termination condition, i.e. the end of the log (see http://doxygen.postgresql.org/xlog_8c.html#a0519e464bfaa79bde3e241e6cff986c7). My question is how does recovery distinguish between the actual end of the log as opposed to a log file corruption (e.g. torn page)?
I'd like to be able to distinguish between a corruption in the log vs. a normal recovery condition if possible.
Thanks.
On Mon, 2012-02-27 at 16:30 -0800, Jameison Martin wrote: > I'd like to get some clarification around an architectural point about > recovery. I see that it is normal to see "unexpected pageaddr" errors > during recovery because of the way Postgres overwrites old log files, > and thus this is taken to be a normal termination condition, i.e. the > end of the log (see > http://doxygen.postgresql.org/xlog_8c.html#a0519e464bfaa79bde3e241e6cff986c7). My question is how does recovery distinguishbetween the actual end of the log as opposed to a log file corruption (e.g. torn page)? > > > I'd like to be able to distinguish between a corruption in the log vs. > a normal recovery condition if possible. If you have a power failure, a torn page in the WAL is expected. Torn pages in the data pages are fixed up using WAL; but WAL doesn't have anything under it to prevent/fix torn pages (unless your filesystem prevents them). Of course, checksums are used to prevent recovery from attempting to play a partial or otherwise corrupt WAL record. What kind of corruption are you trying to detect? Regards, Jeff Davis
i'm not suggesting that Postgres should necessarily be able to repair a corruption in the WAL, my issue is that there is no way of knowing that there was a corruption in the WAL because the normal detection of the end of the WAL is reported the same way as a corruption in the middle of the WAL.
i'd like some unequivocal indication that the WAL was corrupted.
thanks.
From: Jeff Davis <pgsql@j-davis.com>
To: Jameison Martin <jameisonb@yahoo.com>
Cc: "pgsql-general@postgresql.org" <pgsql-general@postgresql.org>
Sent: Wednesday, March 14, 2012 3:55 PM
Subject: Re: [GENERAL] xlog corruption
On Mon, 2012-02-27 at 16:30 -0800, Jameison Martin wrote:
> I'd like to get some clarification around an architectural point about
> recovery. I see that it is normal to see "unexpected pageaddr" errors
> during recovery because of the way Postgres overwrites old log files,
> and thus this is taken to be a normal termination condition, i.e. the
> end of the log (see
> http://doxygen.postgresql.org/xlog_8c.html#a0519e464bfaa79bde3e241e6cff986c7). My question is how does recovery distinguish between the actual end of the log as opposed to a log file corruption (e.g. torn page)?
>
>
> I'd like to be able to distinguish between a corruption in the log vs.
> a normal recovery condition if possible.
If you have a power failure, a torn page in the WAL is expected. Torn
pages in the data pages are fixed up using WAL; but WAL doesn't have
anything under it to prevent/fix torn pages (unless your filesystem
prevents them).
Of course, checksums are used to prevent recovery from attempting to
play a partial or otherwise corrupt WAL record.
What kind of corruption are you trying to detect?
Regards,
Jeff Davis