On Tue, Aug 26, 2025 at 3:44 PM Dimitrios Apostolou <jimis@gmx.net> wrote:
I am storing dumps of a database (pg_dump custom format) in a de-duplicating backup server. Each dump is many terabytes in size, so deduplication is very important. And de-duplication itself is based on rolling checksums which is pretty flexible, it can compensate for blocks moving by some offset.
I suggest looking into pgBackRest, and it's block incremental feature, which sounds similar to what you are doing. But it also does it with parallel processes, and can do things like grab backup files from your replicas, plus a lot of other features.
