Thread: System crash - invalid page header messages in log
One of my systems crashed today and when Postgres started it gave the following warnings. Is this OK? I am going to find which database has these relations and do some checking. It would be nice if the startup wal code gave the database oid also and database version. Thanks Jim select version(); version ------------------------------------------------------------------------------------------PostgreSQL 7.4.2 on i686-pc-linux-gnu,compiled by GCC gcc (GCC) 3.3.3 (Debian 20040401) 2004-09-28 16:10:47 [979] LOG: database system was interrupted at 2004-09-28 16:04:49 EDT 2004-09-28 16:10:47 [979] LOG: checkpoint record is at F4/FE424 2004-09-28 16:10:47 [979] LOG: redo record is at F4/4A94; undo record is at 0/0; shutdown FALSE 2004-09-28 16:10:47 [979] LOG: next transaction ID: 1332030; next OID: 920402645 2004-09-28 16:10:47 [979] LOG: database system was not properly shut down; automatic recovery in progress 2004-09-28 16:10:47 [979] LOG: redo starts at F4/4A94 2004-09-28 16:10:50 [979] WARNING: invalid page header in block 75660 of relation "743151645"; zeroing out page 2004-09-28 16:10:50 [979] WARNING: invalid page header in block 75672 of relation "743151645"; zeroing out page 2004-09-28 16:10:50 [979] WARNING: invalid page header in block 75698 of relation "743151645"; zeroing out page 2004-09-28 16:10:50 [979] WARNING: invalid page header in block 75786 of relation "743151645"; zeroing out page 2004-09-28 16:10:51 [979] WARNING: invalid page header in block 11193 of relation "743151650"; zeroing out page 2004-09-28 16:10:51 [979] WARNING: invalid page header in block 78348 of relation "743151645"; zeroing out page 2004-09-28 16:10:51 [979] WARNING: invalid page header in block 78452 of relation "743151645"; zeroing out page 2004-09-28 16:10:51 [979] WARNING: invalid page header in block 11279 of relation "743151650"; zeroing out page 2004-09-28 16:10:51 [979] WARNING: invalid page header in block 11283 of relation "743151651"; zeroing out page 2004-09-28 16:10:52 [979] WARNING: invalid page header in block 79467 of relation "743151645"; zeroing out page 2004-09-28 16:10:52 [979] WARNING: invalid page header in block 79479 of relation "743151645"; zeroing out page 2004-09-28 16:10:52 [979] WARNING: invalid page header in block 79493 of relation "743151645"; zeroing out page 2004-09-28 16:10:52 [979] WARNING: invalid page header in block 11425 of relation "743151650"; zeroing out page 2004-09-28 16:10:52 [979] WARNING: invalid page header in block 79529 of relation "743151645"; zeroing out page 2004-09-28 16:10:52 [979] WARNING: invalid page header in block 79547 of relation "743151645"; zeroing out page 2004-09-28 16:10:52 [979] WARNING: invalid page header in block 79597 of relation "743151645"; zeroing out page 2004-09-28 16:10:52 [979] WARNING: invalid page header in block 79647 of relation "743151645"; zeroing out page 2004-09-28 16:10:52 [979] WARNING: invalid page header in block 79789 of relation "743151645"; zeroing out page 2004-09-28 16:10:52 [979] WARNING: invalid page header in block 11466 of relation "743151650"; zeroing out page 2004-09-28 16:10:52 [979] WARNING: invalid page header in block 79815 of relation "743151645"; zeroing out page 2004-09-28 16:10:52 [979] WARNING: invalid page header in block 79865 of relation "743151645"; zeroing out page 2004-09-28 16:10:52 [979] WARNING: invalid page header in block 79885 of relation "743151645"; zeroing out page 2004-09-28 16:10:52 [979] WARNING: invalid page header in block 79897 of relation "743151645"; zeroing out page 2004-09-28 16:10:52 [979] WARNING: invalid page header in block 11481 of relation "743151650"; zeroing out page 2004-09-28 16:10:52 [979] WARNING: invalid page header in block 79923 of relation "743151645"; zeroing out page 2004-09-28 16:10:52 [979] WARNING: invalid page header in block 79935 of relation "743151645"; zeroing out page 2004-09-28 16:10:52 [979] WARNING: invalid page header in block 79947 of relation "743151645"; zeroing out page 2004-09-28 16:10:52 [979] WARNING: invalid page header in block 11478 of relation "743151651"; zeroing out page 2004-09-28 16:10:52 [979] LOG: incorrect resource manager data checksum in record at F4/72BEFA0 2004-09-28 16:10:52 [979] LOG: redo done at F4/72BEF3C 2004-09-28 16:11:02 [979] LOG: recycled transaction log file "000000F3000000F9" 2004-09-28 16:11:02 [979] LOG: recycled transaction log file "000000F3000000FA" 2004-09-28 16:11:02 [979] LOG: recycled transaction log file "000000F3000000FB" 2004-09-28 16:11:02 [979] LOG: recycled transaction log file "000000F3000000FC" 2004-09-28 16:11:02 [979] LOG: recycled transaction log file "000000F3000000FD" 2004-09-28 16:11:02 [979] LOG: recycled transaction log file "000000F3000000FE" 2004-09-28 16:11:02 [979] LOG: database system is ready
"Jim Buttafuoco" <jim@contactbda.com> writes:
> One of my systems crashed today and when Postgres started it gave the
> following warnings. Is this OK?
Should theoretically be OK --- all of those pages were overwritten with
valid data from WAL playback. What you need to worry about more is
whether the crash lost any pages that were *not* restored from the log
(because they hadn't been modified lately). I'd suggest running a
pg_dump to see if any bad data turns up.
regards, tom lane
On Tue, Sep 28, 2004 at 04:18:44PM -0400, Jim Buttafuoco wrote: > One of my systems crashed today and when Postgres started it gave the > following warnings. Is this OK? I am going to find which database > has these relations and do some checking. It would be nice if the > startup wal code gave the database oid also and database version. Is this running on XFS? The "invalid page header" messages should not cause you any problem: what happenned is that while recovery was on progress, some pages were found to be invalid. This is no big deal because they can be recovered from XLog (which is why they are zeroed out automatically in the first place). Why would they be corrupted is another matter ... partial writes may cause bad pages, but I think the header is written before the contents when the page is fsync'ed. However, there is some zeroing behaviour on XFS under certain conditions. That could explain the heavy corruption you are seeing. Hopefully no other page should have been zeroed, else it would have had a snapshot on XLog. > 2004-09-28 16:10:52 [979] LOG: incorrect resource manager data checksum in record at F4/72BEFA0 This one may be more problematic ... not sure. Maybe it means that a XLog record wasn't fully written or something. -- Alvaro Herrera (<alvherre[a]dcc.uchile.cl>) "No es bueno caminar con un hombre muerto"
thanks for the reply. I will do a dump of the databases tonight (around 400GB of data) Jim ---------- Original Message ----------- From: Tom Lane <tgl@sss.pgh.pa.us> To: jim@contactbda.com Cc: "pgsql-hackers" <pgsql-hackers@postgresql.org> Sent: Tue, 28 Sep 2004 17:42:04 -0400 Subject: Re: [HACKERS] System crash - invalid page header messages in log > "Jim Buttafuoco" <jim@contactbda.com> writes: > > One of my systems crashed today and when Postgres started it gave the > > following warnings. Is this OK? > > Should theoretically be OK --- all of those pages were overwritten with > valid data from WAL playback. What you need to worry about more is > whether the crash lost any pages that were *not* restored from the log > (because they hadn't been modified lately). I'd suggest running a > pg_dump to see if any bad data turns up. > > regards, tom lane ------- End of Original Message -------