Ok. We didn't find any errors in syslog, in the kern log there were
only some ntpd errors related to apparmor:
...
Dec 19 01:55:45 dallocalbeacondb01c kernel: [960716.006995] audit:
type=1400 audit(1545184545.259:1257): apparmor="DENIED"
operation="open" profile="/usr/sbin/ntpd" name="/usr/local/sbin/"
pid=18444 comm="ntpd" requested_mask="r" denied_mask="r" fsuid=0
ouid=0
...
That said, these are VMs running on ESX hosts with SSD, so it's
certainly possible. We'll check the hosts as well.
For future reference, is there a straight forward way to get all rows
that dont' have any data on a given corrupt page? (rather than
restore to a point in time from a backup)
On Mon, Dec 31, 2018 at 11:57 PM Andrew Gierth
<andrew@tao11.riddles.org.uk> wrote:
>
> >>>>> "Phil" == Phil Hildebrand <phil.hildebrand@moz.com> writes:
>
> Phil> Yeah - There's no sensitive data (it's public domain reviews).
> Phil> I've attached the hexdump
>
> OK. The page is definitely corrupt starting at offset 0x1000 (i.e.
> offset 4kB in the 8kB page), but it's harder than usual to spot because
> (a) the tear is in the middle of a text field and (b) the data in the
> second half of the page is actually from a page of what is presumably a
> different partition of the same table (it has the same row structure,
> but the data is from 2017 not 2018).
>
> The split being on a 4k boundary points the finger at the hardware or
> OS, since pg doesn't care about 4k hardware pages or 4k disk sectors but
> rather does everything in 8k blocks.
>
> --
> Andrew.
--
Phil Hildebrand
Sr. DBE @ Moz
206.696.3413
