On Wed, Jun 12, 2024 at 12:37:46PM -0700, Andres Freund wrote:
> I'm wonder if this isn't going in the wrong direction. We're using CRCs for
> something they're not well suited for in my understanding - and are paying a
> reasonably high price for it, given that even hardware accelerated CRCs aren't
> blazingly fast.
I tend to agree, especially that we should be more concerned about all
bytes after a certain point being garbage than bit flips. (I think we
should also care about bit flips, but I hope those are much less common
than half-written WAL records.)
> With that I perhaps have established that CRC guarantees aren't useful for us.
> But not yet why we should use something else: Given that we already aren't
> relying on hard guarantees, we could instead just use a fast hash like xxh3.
> https://github.com/Cyan4973/xxHash which is fast both for large and small
> amounts of data.
Would it be out of the question to reuse the page checksum code (i.e., an
FNV-1a derivative)? The chart in your link claims that xxh3 is
substantially faster than "FNV64", but I wonder if the latter was
vectorized. I don't know how our CRC-32C implementations (and proposed
implementations) compare, either.
--
nathan
