Re: backup manifests - Mailing list pgsql-hackers

On 11/22/19 1:24 PM, Robert Haas wrote:
> On Tue, Nov 19, 2019 at 4:34 PM David Steele <david@pgmasters.net> wrote:
>> On 11/19/19 5:00 AM, Rushabh Lathia wrote:
>>> My colleague Suraj did testing and noticed the performance impact
>>> with the checksums.   On further testing, he found that specifically with
>>> sha its more of performance impact.
>>
>> We have found that SHA1 adds about 3% overhead when the backup is also
>> compressed (gzip -6), which is what most people want to do.  This
>> percentage goes down even more if the backup is being transferred over a
>> network or to an object store such as S3.
> 
> I don't really understand why your tests and Suraj's tests are showing
> such different results, or how compression plays into it. I tried
> running shasum -a$N lineitem-big.csv on my laptop, where that file
> contains ~70MB of random-looking data whose source I no longer
> remember. Here are the results by algorithm: SHA1, ~25 seconds; SHA224
> or SHA256, ~52 seconds; SHA384 and SHA512, ~39 seconds. Aside from the
> interesting discovery that the algorithms with more bits actually run
> faster on this machine, this seems to show that there's only about a
> ~2x difference between the SHA1 that you used and that I (pretty much
> arbitrarily) used. But Rushabh and Suraj are reporting 43-54%
> overhead, and even if you divide that by two it's a lot more than 3%.
> 
> One possible explanation is that the compression is really slow, and
> so it makes the checksum overhead a smaller percentage of the total.
> Like, if you've already slowed down the backup by 8x, then 24%
> overhead turns into 3% overhead! But I assume that's not the real
> explanation here. 

That's the real explanation here.  Hash calculations run at the same
speed, they just become a smaller portion of the *total* time once
compression (gzip -6) is added.  With something like lz4 hashing will
obviously be a big percentage of the total.

Also consider how much extra latency you get from copying over a
network.  My 3% did not include that but realistically most backups are
running over a network (hopefully).

>> That said, making SHA256 optional seems reasonable.  We decided not to
>> make our SHA1 checksums optional to reduce the test matrix and because
>> parallelism largely addressed performance concerns.
> 
> Just to be clear, I really don't have any objection to using SHA1
> instead of SHA256, or anything else for that matter. I picked the one
> to use out of a hat for the purpose of having a POC quickly; I didn't
> have any intention to insist on that as the final selection. It seems
> likely that anything we pick here will eventually be considered
> obsolete, so I think we need to allow for configurability, but I don't
> have a horse in the game as far as an initial selection goes.

We decided that SHA1 was good enough and there was no need to go up to
SHA256.  What we were interested in was collision rates and what the
chance of getting a false positive were based on the combination of
path, size, and hash.  With SHA1 the chance of a collision was literally
astronomically low (as in the universe would probably end before it
happened, depending on whether you are an expand forever or contract
proponent).

> Except - and this gets back to the previous point - I don't want to
> slow down backups by 40% by default. I wouldn't mind slowing them down
> 3% by default, but 40% is too much overhead. I think we've gotta
> either the overhead of using SHA way down or not use SHA by default.

Maybe -- my take is that the measurements, an uncompressed backup to the
local filesystem, are not a very realistic use case.

However, I'm still fine with leaving the user the option of checksums or
no.  I just wanted to point out that CRCs have their limits so maybe
that's not a great option unless it is properly caveated and perhaps not
the default.

Regards,
-- 
-David
david@pgmasters.net