Re: "PANIC: could not open critical system index 2662" - twice - Mailing list pgsql-general

From Thomas Munro
Subject Re: "PANIC: could not open critical system index 2662" - twice
Date
Msg-id CA+hUKGJ-yWVQGJQiK5XHpD+97Qc533mNr1L=-sOQmCqDYoMB-w@mail.gmail.com
Whole thread Raw
In response to Re: "PANIC: could not open critical system index 2662" - twice  (Evgeny Morozov <postgresql3@realityexists.net>)
Responses Re: "PANIC: could not open critical system index 2662" - twice
Re: "PANIC: could not open critical system index 2662" - twice
List pgsql-general
On Sat, May 6, 2023 at 9:58 PM Evgeny Morozov
<postgresql3@realityexists.net> wrote:
> Right - I should have realised that! base/1414389/2662 is indeed all
> nulls, 32KB of them. I included the file anyway in
> https://objective.realityexists.net/temp/pgstuff2.zip

OK so it's not just page 0, you have 32KB or 4 pages of all zeroes.
That's the expected length of that relation when copied from the
initial template, and consistent with the pg_waldump output (it uses
FPIs to copy blocks 0-3).  We can't see the block contents but we know
that block 2 definitely is not all zeroes at that point because there
are various modifications to it, which not only write non-zeroes but
must surely have required a sane page 0.

So it does indeed look like something unknown has replaced 32KB of
data with 32KB of zeroes underneath us.  Are there more non-empty
files that are all-zeroes?  Something like this might find them:

for F in base/1414389/*
do
  if [ -s $F ] && ! xxd -p $F | grep -qEv '^(00)*$' > /dev/null
  then
    echo $F
  fi
done



pgsql-general by date:

Previous
From: Evgeny Morozov
Date:
Subject: Re: "PANIC: could not open critical system index 2662" - twice
Next
From: Marc Millas
Date:
Subject: Re: Death postgres