Re: Silent data corruption in PostgreSQL 17 - how to detect it proactively? - Mailing list pgsql-general

From Pawel Kudzia
Subject Re: Silent data corruption in PostgreSQL 17 - how to detect it proactively?
Date
Msg-id CAJYBUS8UQboD_cNC9qzPSai1iyY4m1eExy1GRnJEJfZX0p4Psw@mail.gmail.com
Whole thread
In response to Re: Silent data corruption in PostgreSQL 17 - how to detect it proactively?  (Merlin Moncure <mmoncure@gmail.com>)
List pgsql-general
On Tue, Sep 16, 2025 at 6:41 PM Merlin Moncure <mmoncure@gmail.com> wrote:
>
> On Tue, Sep 16, 2025 at 7:25 AM Vladlen Popolitov <v.popolitov@postgrespro.ru> wrote:
>>
>> Checksum calculation takes ~0.5% of query time, it is not bottleneck
>> in PostgreSQL.
>
>
> I consider checksums=on to be a mandatory setting.  Often, these types of things are not bugs in postgres itself, but
bugsin storage, the underlying operating system, or extensions.  Checksums can and will protect you, and may even bring
youclose to the thing causing the corruption.  Given that your replica is ok, this very much smells like a similar type
ofissue. 
>
> In a prior case, I was using pl/sh to load data to the database with 'copy', and for what I believe to be o/s issues,
wasgetting corruption.  Enabling checksums completely addressed the source of the damage.  Turn them on! 
>


i'd like to add a note that might help others - at the end - it was
not PostgreSQL, but a bug in underlying KVM when snapshots were
active. it hit others as well and at the time of writing is not
resolved yet - https://gitlab.com/qemu-project/qemu/-/issues/3273

thanks a lot for help and suggestions!




--
regards,
Pawel Kudzia



pgsql-general by date:

Previous
From: Linus Heckemann
Date:
Subject: Re: Can we get sha* function over text, that could be used in index?
Next
From: "Peter J. Holzer"
Date:
Subject: Re: Can we get sha* function over text, that could be used in index?