Re: PostgreSQL's handling of fsync() errors is unsafe and risks data loss at least on XFS - Mailing list pgsql-hackers

From Bruce Momjian
Subject Re: PostgreSQL's handling of fsync() errors is unsafe and risks data loss at least on XFS
Date
Msg-id 20180404175103.GA12574@momjian.us
Whole thread Raw
In response to Re: PostgreSQL's handling of fsync() errors is unsafe and risks data loss at least on XFS  (Craig Ringer <craig@2ndquadrant.com>)
List pgsql-hackers
On Wed, Apr  4, 2018 at 11:23:51PM +0800, Craig Ringer wrote:
> On 4 April 2018 at 21:49, Bruce Momjian <bruce@momjian.us> wrote:
>     I can understand why kernel developers don't want to keep failed sync
>     buffers in memory, and once they are gone we lose reporting of their
>     failure.  Also, if the kernel is going to not retry the syncs, how long
>     should it keep reporting the sync failure?
> 
> Ideally until the app tells it not to.
> 
> But there's no standard API for that.

You would almost need an API that registers _before_ the failure that
you care about sync failures, and that you plan to call fsync() to
gather such information.  I am not sure how you would allow more than
the first fsync() to see the failure unless you added _another_ API to
clear the fsync failure, but I don't see the point since the first
fsync() might call that clear function.  How many applications are going
to know there is _another_ application that cares about the failure? Not
many.

> Currently we kind of hope the kernel will deal with figuring out which
> is which and retrying. Turns out it doesn't do that so much, and I
> don't think the reasons for that are wholly unreasonable. We may have
> been asking too much.

Agreed.

>     Our only option might be to tell administrators to closely watch for
>     kernel write failure messages, and then restore or failover.  :-(
> 
> Speaking of, there's not necessarily any lost page write error in the logs
> AFAICS. My tests often just show "Buffer I/O error on device dm-0, logical
> block 59393" or the like.

I assume that is the kernel logs.  I am thinking the kernel logs have to
be monitored, but how many administrators do that?  The other issue I
think you are pointing out is how is the administrator going to know
this is a Postgres file?  I guess any sync error to a device that
contains Postgres has to assume Postgres is corrupted.  :-(

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +


pgsql-hackers by date:

Previous
From: Tomas Vondra
Date:
Subject: Re: some last patches breaks plan cache
Next
From: Tom Lane
Date:
Subject: Re: Add support for printing/reading MergeAction nodes