Thread: page is uninitialized --- fixing

page is uninitialized --- fixing

From
Simon Riggs
Date:
A couple of people in recent years have had a problem with "page X is
uninitialised -- fixing" messages.

I have a case now with 569357 consecutive pages that required fixing in
pg_attribute. We looked at pages by hand and they really are
uninitialised, but otherwise what we would expect for size, name etc..

Clearly this is way too many pages to be easily explainable.

Historically, this type of error has occurred mostly on servers that
have been through a recovery, so I have investigated it with that in
mind as a potential error source. Nothing found on that score, though
rsync is in use, as before.

One factor here is that temp tables are very heavily used. The size of
the pg_attribute table is *roughly* what I would expect, given the
frequency of temp table creation, numbers of cols used and lack of
vacuum. 

The server did have non-ECC memory and there have been a few other
memory issues, but I'm still a little worried by this.

A completely separate client has twice had corrupted indexes on pg_class
in last 6 months, again a heavy user of temp tables. I've looked for
issues around the idea of all-temp catalog pages causing an problem, but
not seen anything as yet.

Any issues or ideas worth investigating?

-- Simon Riggs           www.2ndQuadrant.comPostgreSQL Training, Services and Support



Re: page is uninitialized --- fixing

From
Tom Lane
Date:
Simon Riggs <simon@2ndQuadrant.com> writes:
> A couple of people in recent years have had a problem with "page X is
> uninitialised -- fixing" messages.

> I have a case now with 569357 consecutive pages that required fixing in
> pg_attribute. We looked at pages by hand and they really are
> uninitialised, but otherwise what we would expect for size, name etc..

What PG version?
        regards, tom lane


Re: page is uninitialized --- fixing

From
Simon Riggs
Date:
On Tue, 2009-06-09 at 13:54 -0400, Tom Lane wrote:
> Simon Riggs <simon@2ndQuadrant.com> writes:
> > A couple of people in recent years have had a problem with "page X is
> > uninitialised -- fixing" messages.
> 
> > I have a case now with 569357 consecutive pages that required fixing in
> > pg_attribute. We looked at pages by hand and they really are
> > uninitialised, but otherwise what we would expect for size, name etc..
> 
> What PG version?

PG 8.3.7 on CentOS. Other client mentioned is 8.3 on SuSE.

-- Simon Riggs           www.2ndQuadrant.comPostgreSQL Training, Services and Support



Re: page is uninitialized --- fixing

From
Tom Lane
Date:
Simon Riggs <simon@2ndQuadrant.com> writes:
> A couple of people in recent years have had a problem with "page X is
> uninitialised -- fixing" messages.

> I have a case now with 569357 consecutive pages that required fixing in
> pg_attribute. We looked at pages by hand and they really are
> uninitialised, but otherwise what we would expect for size, name etc..

> Clearly this is way too many pages to be easily explainable.

It's probably too late to tell now, but I wonder if those pages actually
existed or were just a "hole" in the file.  A perhaps-plausible
mechanism for them to appear is that the FSM spits out some ridiculously
large page number as being the next place to insert something into
pg_attribute, the system plops down a new tuple into that page, and
behold you have a large hole that reads as zeroes.

Another interesting question is whether the range began or ended at a
1GB segment boundary, in which case something in or around the
segmenting logic could be at fault.  (Hmm ... actually 1GB is only
131072 pages anyway, so your "hole" definitely spanned several segments.
That seems like the next place to look.)
        regards, tom lane


Re: page is uninitialized --- fixing

From
Simon Riggs
Date:
On Tue, 2009-06-09 at 16:17 -0400, Tom Lane wrote:
> Simon Riggs <simon@2ndQuadrant.com> writes:
> > A couple of people in recent years have had a problem with "page X is
> > uninitialised -- fixing" messages.
> 
> > I have a case now with 569357 consecutive pages that required fixing in
> > pg_attribute. We looked at pages by hand and they really are
> > uninitialised, but otherwise what we would expect for size, name etc..
> 
> > Clearly this is way too many pages to be easily explainable.
> 
> It's probably too late to tell now, but I wonder if those pages actually
> existed or were just a "hole" in the file.  A perhaps-plausible
> mechanism for them to appear is that the FSM spits out some ridiculously
> large page number as being the next place to insert something into
> pg_attribute, the system plops down a new tuple into that page, and
> behold you have a large hole that reads as zeroes.
> 
> Another interesting question is whether the range began or ended at a
> 1GB segment boundary, in which case something in or around the
> segmenting logic could be at fault.  (Hmm ... actually 1GB is only
> 131072 pages anyway, so your "hole" definitely spanned several segments.
> That seems like the next place to look.)

The "hole" started about 0.75GB in file 0 and spanned 4 complete 1GB
segments before records started again in file 5. The "hole" segments
were all 1GB in size, and the pages either size of the hole were
undamaged.

A corrupt record of a block number would do this in XLogReadBuffer() if
we had full page writes enabled. But it would have to be corrupt between
setting it correctly and the CRC check on the WAL record. Which is a
fairly small window of believability.

Should there be a sanity check on how far a relation can be extended in
recovery?

Not sure if that would work with normal mode ReadBuffer() - it should
fail somewhere in smgr or in bufmgr.

-- Simon Riggs           www.2ndQuadrant.comPostgreSQL Training, Services and Support



Re: page is uninitialized --- fixing

From
Tom Lane
Date:
Simon Riggs <simon@2ndQuadrant.com> writes:
> A corrupt record of a block number would do this in XLogReadBuffer() if
> we had full page writes enabled. But it would have to be corrupt between
> setting it correctly and the CRC check on the WAL record. Which is a
> fairly small window of believability.

> Should there be a sanity check on how far a relation can be extended in
> recovery?

As you say, that's not a highly believable theory.  I'd prefer not to
put an arbitrary limit into the recovery code unless we can positively
pin this down as the source of the problem.

Is there any particular reason to suppose that the empty pages appeared
during a crash recovery?

Have you read through md.c?  I seem to recall there are some slightly
squirrelly choices made there about segment-extension behavior.  Maybe
it could've done the wrong thing here during normal operation.
        regards, tom lane


Re: page is uninitialized --- fixing

From
Simon Riggs
Date:
On Tue, 2009-06-09 at 17:28 -0400, Tom Lane wrote:
> Simon Riggs <simon@2ndQuadrant.com> writes:
> > A corrupt record of a block number would do this in XLogReadBuffer() if
> > we had full page writes enabled. But it would have to be corrupt between
> > setting it correctly and the CRC check on the WAL record. Which is a
> > fairly small window of believability.
> 
> > Should there be a sanity check on how far a relation can be extended in
> > recovery?
> 
> As you say, that's not a highly believable theory.  I'd prefer not to
> put an arbitrary limit into the recovery code unless we can positively
> pin this down as the source of the problem.
> 
> Is there any particular reason to suppose that the empty pages appeared
> during a crash recovery?

Probably because my Rorschach tests all look like database recoveries.

> Have you read through md.c?  I seem to recall there are some slightly
> squirrelly choices made there about segment-extension behavior.  Maybe
> it could've done the wrong thing here during normal operation.

Yes, but will do again if you think I should check.

-- Simon Riggs           www.2ndQuadrant.comPostgreSQL Training, Services and Support



Re: page is uninitialized --- fixing

From
Simon Riggs
Date:
On Tue, 2009-06-09 at 17:28 -0400, Tom Lane wrote:

> Is there any particular reason to suppose that the empty pages appeared
> during a crash recovery?
> 
> Have you read through md.c?  I seem to recall there are some slightly
> squirrelly choices made there about segment-extension behavior.  Maybe
> it could've done the wrong thing here during normal operation.

The only way I can see this happening is

1. backup: pg_start_backup()
2. user1: creates temp table
3. user2: Vacuum causes partial truncation of table
4. backup: copies table's files
5. backup: pg_stop_backup()

When we replay this, we do so with a base backup that starts recovering
at XLogRecPtr of start_backup(), but using a file copied at a later
XLogRecPtr.

I don't see any way thru md.c that would allow reading a block beyond
the end of the currently existing file (except in recovery).

I'm fairly certain that this didn't happen because the table in question
was pg_attribute and the main problem was that it had not been vacuumed
for a very long time. Which leaves me perplexed.

An erroneous value from file seek would lead to this though, so perhaps
that is the easiest explanation: damage to file metadata in memory.

-- Simon Riggs           www.2ndQuadrant.comPostgreSQL Training, Services and Support