Thread: Initdb --data-checksums by default
Attachment
På onsdag 20. april 2016 kl. 10:33:14, skrev Alex Ignatov <a.ignatov@postgrespro.ru>:
On 20.04.2016 11:29, Devrim Gündüz wrote:
> Hi,
>
> On Wed, 2016-04-20 at 10:43 +0300, Alex Ignatov wrote:
>> Today in Big Data epoch silent data corruption becoming more and more
>> issue to afraid of. With uncorrectable read error rate ~ 10^-15 on
>> multiterabyte disk bit rot is the real issue.
>> I think that today checksumming data must be mandatory set by default.
>> Only if someone doesn't care about his data he can manually turn this
>> option off.
>>
>> What do you think about defaulting --data-checksums in initdb?
> I think this should be discussed in -hackers, right?
>
> Regards,
May be you right but i want to know what people think about it before
i'll write to hackers.
-1 on changing the default.
10^15 ~= 1000 TB, which isn't very common yet. Those having it probably are aware of the risk and have enabled checksums already.
-- 
Andreas Joseph Krogh
CTO / Partner - Visena AS
Mobile: +47 909 56 963
Attachment
On 20.04.2016 11:40, Andreas Joseph Krogh wrote:
It is per bit not bytes. So it is ~100 TB. We working with some enterprise who have WALs creation rate ~ 4GB per min - so it is only max 100 days before you get bit rotted and have probability to get silent data corruption.På onsdag 20. april 2016 kl. 10:33:14, skrev Alex Ignatov <a.ignatov@postgrespro.ru>:
On 20.04.2016 11:29, Devrim Gündüz wrote:
> Hi,
>
> On Wed, 2016-04-20 at 10:43 +0300, Alex Ignatov wrote:
>> Today in Big Data epoch silent data corruption becoming more and more
>> issue to afraid of. With uncorrectable read error rate ~ 10^-15 on
>> multiterabyte disk bit rot is the real issue.
>> I think that today checksumming data must be mandatory set by default.
>> Only if someone doesn't care about his data he can manually turn this
>> option off.
>>
>> What do you think about defaulting --data-checksums in initdb?
> I think this should be discussed in -hackers, right?
>
> Regards,
May be you right but i want to know what people think about it before
i'll write to hackers.-1 on changing the default.10^15 ~= 1000 TB, which isn't very common yet. Those having it probably are aware of the risk and have enabled checksums already.--Andreas Joseph KroghCTO / Partner - Visena ASMobile: +47 909 56 963
Also don't forget that it is theoretical limit and Google tells us that HDD and SSD is not as reliable as manufactures tell. So this 10^-15 can easily be much higher.
-- Alex Ignatov Postgres Professional: http://www.postgrespro.com The Russian Postgres Company
Attachment
På onsdag 20. april 2016 kl. 11:02:31, skrev Alex Ignatov <a.ignatov@postgrespro.ru>:
On 20.04.2016 11:40, Andreas Joseph Krogh wrote:It is per bit not bytes. So it is ~100 TB. We working with some enterprise who have WALs creation rate ~ 4GB per min - so it is only max 100 days before you get bit rotted and have probability to get silent data corruption.På onsdag 20. april 2016 kl. 10:33:14, skrev Alex Ignatov <a.ignatov@postgrespro.ru>:
On 20.04.2016 11:29, Devrim Gündüz wrote:
> Hi,
>
> On Wed, 2016-04-20 at 10:43 +0300, Alex Ignatov wrote:
>> Today in Big Data epoch silent data corruption becoming more and more
>> issue to afraid of. With uncorrectable read error rate ~ 10^-15 on
>> multiterabyte disk bit rot is the real issue.
>> I think that today checksumming data must be mandatory set by default.
>> Only if someone doesn't care about his data he can manually turn this
>> option off.
>>
>> What do you think about defaulting --data-checksums in initdb?
> I think this should be discussed in -hackers, right?
>
> Regards,
May be you right but i want to know what people think about it before
i'll write to hackers.-1 on changing the default.10^15 ~= 1000 TB, which isn't very common yet. Those having it probably are aware of the risk and have enabled checksums already.--Andreas Joseph KroghCTO / Partner - Visena ASMobile: +47 909 56 963
Also don't forget that it is theoretical limit and Google tells us that HDD and SSD is not as reliable as manufactures tell. So this 10^-15 can easily be much higher.
Ok, but still - the case you're describing isn't the common-case for PG-users. Enterprises like that certainly chould use --data-checksums, I'm not arguing against that, just that it shouldn't be the default-setting.
-- 
Andreas Joseph Krogh
CTO / Partner - Visena AS
Mobile: +47 909 56 963
Attachment
On 20.04.2016 12:10, Andreas Joseph Krogh wrote:
Why do you think that common pg-users doesn't care about their data? Also why do we have wal_level=minimal fsync=on and other stuff?På onsdag 20. april 2016 kl. 11:02:31, skrev Alex Ignatov <a.ignatov@postgrespro.ru>:On 20.04.2016 11:40, Andreas Joseph Krogh wrote:It is per bit not bytes. So it is ~100 TB. We working with some enterprise who have WALs creation rate ~ 4GB per min - so it is only max 100 days before you get bit rotted and have probability to get silent data corruption.På onsdag 20. april 2016 kl. 10:33:14, skrev Alex Ignatov <a.ignatov@postgrespro.ru>:
On 20.04.2016 11:29, Devrim Gündüz wrote:
> Hi,
>
> On Wed, 2016-04-20 at 10:43 +0300, Alex Ignatov wrote:
>> Today in Big Data epoch silent data corruption becoming more and more
>> issue to afraid of. With uncorrectable read error rate ~ 10^-15 on
>> multiterabyte disk bit rot is the real issue.
>> I think that today checksumming data must be mandatory set by default.
>> Only if someone doesn't care about his data he can manually turn this
>> option off.
>>
>> What do you think about defaulting --data-checksums in initdb?
> I think this should be discussed in -hackers, right?
>
> Regards,
May be you right but i want to know what people think about it before
i'll write to hackers.-1 on changing the default.10^15 ~= 1000 TB, which isn't very common yet. Those having it probably are aware of the risk and have enabled checksums already.--Andreas Joseph KroghCTO / Partner - Visena ASMobile: +47 909 56 963
Also don't forget that it is theoretical limit and Google tells us that HDD and SSD is not as reliable as manufactures tell. So this 10^-15 can easily be much higher.Ok, but still - the case you're describing isn't the common-case for PG-users. Enterprises like that certainly chould use --data-checksums, I'm not arguing against that, just that it shouldn't be the default-setting.--Andreas Joseph KroghCTO / Partner - Visena ASMobile: +47 909 56 963
-- Alex Ignatov Postgres Professional: http://www.postgrespro.com The Russian Postgres Company
Attachment
På onsdag 20. april 2016 kl. 11:22:33, skrev Alex Ignatov <a.ignatov@postgrespro.ru>:
[snip]
Why do you think that common pg-users doesn't care about their data?
Did I say that?
Also why do we have wal_level=minimal fsync=on and other stuff?
To make certain garantees that data is by default durable.
What I'm saying is that everything is a compromise, cost/benefit. The universe might explode tomorrow but the chances are slim, so no use preparing for it.
Those caring enough probably use checksums, battery-packed RAID etc.
-- 
Andreas Joseph Krogh
CTO / Partner - Visena AS
Mobile: +47 909 56 963
Attachment
On 20.04.2016 12:27, Andreas Joseph Krogh wrote:
På onsdag 20. april 2016 kl. 11:22:33, skrev Alex Ignatov <a.ignatov@postgrespro.ru>:[snip]Why do you think that common pg-users doesn't care about their data?Did I say that?Also why do we have wal_level=minimal fsync=on and other stuff?To make certain garantees that data is by default durable.What I'm saying is that everything is a compromise, cost/benefit. The universe might explode tomorrow but the chances are slim, so no use preparing for it.Those caring enough probably use checksums, battery-packed RAID etc.--Andreas Joseph KroghCTO / Partner - Visena ASMobile: +47 909 56 963
Wal is not durable without crc in it. But we have by default crc in wal and not in data files.
-- Alex Ignatov Postgres Professional: http://www.postgrespro.com The Russian Postgres Company