On Tue, Jan 10, 2012 at 8:04 AM, Heikki Linnakangas
<heikki.linnakangas@enterprisedb.com> wrote:
> On 10.01.2012 02:12, Jim Nasby wrote:
>>
>> Filesystem CRCs very likely will not happen to data that's in the cache.
>> For some users, that's a huge amount of data to leave un-protected.
>
>
> You can repeat that argument ad infinitum. Even if the CRC covers all the
> pages in the OS buffer cache, it still doesn't cover the pages in the
> shared_buffers, CPU caches, in-transit from one memory bank to another etc.
> You have to draw the line somewhere, and it seems reasonable to draw it
> where the data moves between long-term storage, ie. disk, and RAM.
We protect each change with a CRC when we write WAL, so doing the same
thing doesn't sound entirely unreasonable, especially if your database
fits in RAM and we aren't likely to be doing I/O anytime soon. The
long term storage argument may no longer apply in a world with very
large memory.
The question is, when exactly would we check the checksum? When we
lock the block, when we pin it? We certainly can't do it on every
access to the block since we don't even track where that happens in
the code.
I think we could add an option to check the checksum immediately after
we pin a block for the first time but it would be very expensive and
sounds like we're re-inventing hardware or OS features again. Work on
50% performance drain, as an estimate.
That is a level of protection no other DBMS offers, so that is either
an advantage or a warning. Jim, if you want this, please do the
research and work out what the probability of losing shared buffer
data in your ECC RAM really is so we are doing it for quantifiable
reasons (via old Google memory academic paper) and to verify that the
cost/benefit means you would actually use it if we built it. Research
into requirements is at least as important and time consuming as
research on possible designs.
--
Simon Riggs http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
