Thread: page is uninitialized --- fixing
A couple of people in recent years have had a problem with "page X is uninitialised -- fixing" messages. I have a case now with 569357 consecutive pages that required fixing in pg_attribute. We looked at pages by hand and they really are uninitialised, but otherwise what we would expect for size, name etc.. Clearly this is way too many pages to be easily explainable. Historically, this type of error has occurred mostly on servers that have been through a recovery, so I have investigated it with that in mind as a potential error source. Nothing found on that score, though rsync is in use, as before. One factor here is that temp tables are very heavily used. The size of the pg_attribute table is *roughly* what I would expect, given the frequency of temp table creation, numbers of cols used and lack of vacuum. The server did have non-ECC memory and there have been a few other memory issues, but I'm still a little worried by this. A completely separate client has twice had corrupted indexes on pg_class in last 6 months, again a heavy user of temp tables. I've looked for issues around the idea of all-temp catalog pages causing an problem, but not seen anything as yet. Any issues or ideas worth investigating? -- Simon Riggs www.2ndQuadrant.comPostgreSQL Training, Services and Support
Simon Riggs <simon@2ndQuadrant.com> writes:
> A couple of people in recent years have had a problem with "page X is
> uninitialised -- fixing" messages.
> I have a case now with 569357 consecutive pages that required fixing in
> pg_attribute. We looked at pages by hand and they really are
> uninitialised, but otherwise what we would expect for size, name etc..
What PG version?
regards, tom lane
On Tue, 2009-06-09 at 13:54 -0400, Tom Lane wrote: > Simon Riggs <simon@2ndQuadrant.com> writes: > > A couple of people in recent years have had a problem with "page X is > > uninitialised -- fixing" messages. > > > I have a case now with 569357 consecutive pages that required fixing in > > pg_attribute. We looked at pages by hand and they really are > > uninitialised, but otherwise what we would expect for size, name etc.. > > What PG version? PG 8.3.7 on CentOS. Other client mentioned is 8.3 on SuSE. -- Simon Riggs www.2ndQuadrant.comPostgreSQL Training, Services and Support
Simon Riggs <simon@2ndQuadrant.com> writes:
> A couple of people in recent years have had a problem with "page X is
> uninitialised -- fixing" messages.
> I have a case now with 569357 consecutive pages that required fixing in
> pg_attribute. We looked at pages by hand and they really are
> uninitialised, but otherwise what we would expect for size, name etc..
> Clearly this is way too many pages to be easily explainable.
It's probably too late to tell now, but I wonder if those pages actually
existed or were just a "hole" in the file. A perhaps-plausible
mechanism for them to appear is that the FSM spits out some ridiculously
large page number as being the next place to insert something into
pg_attribute, the system plops down a new tuple into that page, and
behold you have a large hole that reads as zeroes.
Another interesting question is whether the range began or ended at a
1GB segment boundary, in which case something in or around the
segmenting logic could be at fault. (Hmm ... actually 1GB is only
131072 pages anyway, so your "hole" definitely spanned several segments.
That seems like the next place to look.)
regards, tom lane
On Tue, 2009-06-09 at 16:17 -0400, Tom Lane wrote: > Simon Riggs <simon@2ndQuadrant.com> writes: > > A couple of people in recent years have had a problem with "page X is > > uninitialised -- fixing" messages. > > > I have a case now with 569357 consecutive pages that required fixing in > > pg_attribute. We looked at pages by hand and they really are > > uninitialised, but otherwise what we would expect for size, name etc.. > > > Clearly this is way too many pages to be easily explainable. > > It's probably too late to tell now, but I wonder if those pages actually > existed or were just a "hole" in the file. A perhaps-plausible > mechanism for them to appear is that the FSM spits out some ridiculously > large page number as being the next place to insert something into > pg_attribute, the system plops down a new tuple into that page, and > behold you have a large hole that reads as zeroes. > > Another interesting question is whether the range began or ended at a > 1GB segment boundary, in which case something in or around the > segmenting logic could be at fault. (Hmm ... actually 1GB is only > 131072 pages anyway, so your "hole" definitely spanned several segments. > That seems like the next place to look.) The "hole" started about 0.75GB in file 0 and spanned 4 complete 1GB segments before records started again in file 5. The "hole" segments were all 1GB in size, and the pages either size of the hole were undamaged. A corrupt record of a block number would do this in XLogReadBuffer() if we had full page writes enabled. But it would have to be corrupt between setting it correctly and the CRC check on the WAL record. Which is a fairly small window of believability. Should there be a sanity check on how far a relation can be extended in recovery? Not sure if that would work with normal mode ReadBuffer() - it should fail somewhere in smgr or in bufmgr. -- Simon Riggs www.2ndQuadrant.comPostgreSQL Training, Services and Support
Simon Riggs <simon@2ndQuadrant.com> writes:
> A corrupt record of a block number would do this in XLogReadBuffer() if
> we had full page writes enabled. But it would have to be corrupt between
> setting it correctly and the CRC check on the WAL record. Which is a
> fairly small window of believability.
> Should there be a sanity check on how far a relation can be extended in
> recovery?
As you say, that's not a highly believable theory. I'd prefer not to
put an arbitrary limit into the recovery code unless we can positively
pin this down as the source of the problem.
Is there any particular reason to suppose that the empty pages appeared
during a crash recovery?
Have you read through md.c? I seem to recall there are some slightly
squirrelly choices made there about segment-extension behavior. Maybe
it could've done the wrong thing here during normal operation.
regards, tom lane
On Tue, 2009-06-09 at 17:28 -0400, Tom Lane wrote: > Simon Riggs <simon@2ndQuadrant.com> writes: > > A corrupt record of a block number would do this in XLogReadBuffer() if > > we had full page writes enabled. But it would have to be corrupt between > > setting it correctly and the CRC check on the WAL record. Which is a > > fairly small window of believability. > > > Should there be a sanity check on how far a relation can be extended in > > recovery? > > As you say, that's not a highly believable theory. I'd prefer not to > put an arbitrary limit into the recovery code unless we can positively > pin this down as the source of the problem. > > Is there any particular reason to suppose that the empty pages appeared > during a crash recovery? Probably because my Rorschach tests all look like database recoveries. > Have you read through md.c? I seem to recall there are some slightly > squirrelly choices made there about segment-extension behavior. Maybe > it could've done the wrong thing here during normal operation. Yes, but will do again if you think I should check. -- Simon Riggs www.2ndQuadrant.comPostgreSQL Training, Services and Support
On Tue, 2009-06-09 at 17:28 -0400, Tom Lane wrote: > Is there any particular reason to suppose that the empty pages appeared > during a crash recovery? > > Have you read through md.c? I seem to recall there are some slightly > squirrelly choices made there about segment-extension behavior. Maybe > it could've done the wrong thing here during normal operation. The only way I can see this happening is 1. backup: pg_start_backup() 2. user1: creates temp table 3. user2: Vacuum causes partial truncation of table 4. backup: copies table's files 5. backup: pg_stop_backup() When we replay this, we do so with a base backup that starts recovering at XLogRecPtr of start_backup(), but using a file copied at a later XLogRecPtr. I don't see any way thru md.c that would allow reading a block beyond the end of the currently existing file (except in recovery). I'm fairly certain that this didn't happen because the table in question was pg_attribute and the main problem was that it had not been vacuumed for a very long time. Which leaves me perplexed. An erroneous value from file seek would lead to this though, so perhaps that is the easiest explanation: damage to file metadata in memory. -- Simon Riggs www.2ndQuadrant.comPostgreSQL Training, Services and Support