Re: Online verification of checksums - Mailing list pgsql-hackers

From David Steele
Subject Re: Online verification of checksums
Date
Msg-id deb9ef07-d50a-7073-5eaa-9626a843bc70@pgmasters.net
Whole thread Raw
In response to Re: Online verification of checksums  (Stephen Frost <sfrost@snowman.net>)
List pgsql-hackers
On 11/30/20 9:27 AM, Stephen Frost wrote:
> Greetings,
> 
> * Michael Paquier (michael@paquier.xyz) wrote:
>> On Fri, Nov 27, 2020 at 11:15:27AM -0500, Stephen Frost wrote:
>>> * Magnus Hagander (magnus@hagander.net) wrote:
>>>> On Thu, Nov 26, 2020 at 8:42 AM Michael Paquier <michael@paquier.xyz> wrote:
>>>>> But here the checksum is broken, so while the offset is something we
>>>>> can rely on how do you make sure that the LSN is fine?  A broken
>>>>> checksum could perfectly mean that the LSN is actually *not* fine if
>>>>> the page header got corrupted.
>>>
>>> Of course that could be the case, but it gets to be a smaller and
>>> smaller chance by checking that the LSN read falls within reasonable
>>> bounds.
>>
>> FWIW, I find that scary.
> 
> There's ultimately different levels of 'scary' and the risk here that
> something is actually wrong following these checks strikes me as being
> on the same order as random bits being flipped in the page and still
> getting a valid checksum (which is entirely possible with our current
> checksum...), or maybe even less.  

I would say a lot less. First you'd need to corrupt one of the eight 
bytes that make up the LSN (pretty likely since corruption will probably 
affect the entire block) and then it would need to be updated to a value 
that falls within the current backup range, a 1 in 16 million chance if 
a terabyte of WAL is generated during the backup. Plus, the corruption 
needs to happen during the backup since we are going to check for that, 
and the corrupted LSN needs to be ascending, and the LSN originally read 
needs to be within the backup range (another 1 in 16 million chance) 
since pages written before the start backup checkpoint should not be torn.

So as far as I can see there are more likely to be false negatives from 
the checksum itself.

It would also be easy to add a few rounds of checks, i.e. test if the 
LSN ascends but stays in the backup LSN range N times.

Honestly, I'm much more worried about corruption zeroing the entire 
page. I don't know how likely that is, but I know none of our proposed 
solutions would catch it.

Andres, since you brought this issue up originally perhaps you'd like to 
weigh in?

Regards,
-- 
-David
david@pgmasters.net



pgsql-hackers by date:

Previous
From: Masahiko Sawada
Date:
Subject: Re: Add Information during standby recovery conflicts
Next
From: Alvaro Herrera
Date:
Subject: Re: error_severity of brin work item