Thread: Postgres, fsync, and OSs (specifically linux)

Postgres, fsync, and OSs (specifically linux)

From
Andres Freund
Date:
Hi,

I thought I'd send this separately from [0] as the issue has become more
general than what was mentioned in that thread, and it went off into
various weeds.

I went to LSF/MM 2018 to discuss [0] and related issues. Overall I'd say
it was a very productive discussion.  I'll first try to recap the
current situation, updated with knowledge I gained. Secondly I'll try to
discuss the kernel changes that seem to have been agreed upon. Thirdly
I'll try to sum up what postgres needs to change.

== Current Situation ==

The fundamental problem is that postgres assumed that any IO error would
be reported at fsync time, and that the error would be reported until
resolved. That's not true in several operating systems, linux included.

There's various judgement calls leading to the current OS (specifically
linux, but the concerns are similar in other OSs) behaviour:

- By the time IO errors are treated as fatal, it's unlikely that plain
  retries attempting to write exactly the same data are going to
  succeed. There are retries on several layers. Some cases would be
  resolved by overwriting a larger amount (so device level remapping
  functionality can mask dead areas), but plain retries aren't going to
  get there if they didn't the first time round.
- Retaining all the data necessary for retries would make it quite
  possible to turn IO errors on some device into out of memory
  errors. This is true to a far lesser degree if only enough information
  were to be retained to (re-)report an error, rather than actually
  retry the write.
- Continuing to re-report an error after one fsync() failed would make
  it hard to recover from that fact. There'd need to be a way to "clear"
  a persistent error bit, and that'd obviously be outside of posix.
- Some other databases use direct-IO and thus these paths haven't been
  exercised under fire that much.
- Actually marking files as persistently failed would require filesystem
  changes, and filesystem metadata IO, far from guaranteed in failure
  scenarios.

Before linux v4.13 errors in kernel writeback would be reported at most
once, without a guarantee that that'd happen (IIUC memory pressure could
lead to the relevant information being evicted) - but it was pretty
likely.  After v4.13 (see https://lwn.net/Articles/724307/) errors are
reported exactly once to all open file descriptors for a file with an
error - but never for files that have been opened after the error
occurred.

It's worth to note that on linux it's not well defined what contents one
would read after a writeback error. IIUC xfs will mark the pagecache
contents that triggered an error as invalid, triggering a re-read from
the underlying storage (thus either failing or returning old but
persistent contents). Whereas some other filesystems (among them ext4 I
believe) retain the modified contents of the page cache, but marking it
as clean (thereby returning new contents until the page cache contents
are evicted).

Some filesystems (prominently NFS in many configurations) perform an
implicit fsync when closing the file. While postgres checks for an error
of close() and reports it, we don't treat it as fatal. It's worth to
note that by my reading this means that an fsync error at close() will
*not* be re-reported by the time an explicit fsync() is issued. It also
means that we'll not react properly to the possible ENOSPC errors that
may be reported at close() for NFS.  At least the latter isn't just the
case in linux.

Proposals for how postgres could deal with this included using syncfs(2)
- but that turns out not to work at all currently, because syncfs()
basically wouldn't return any file-level errors. It'd also imply
superflously flushing temporary files etc.

The second major type of proposal was using direct-IO. That'd generally
be a desirable feature, but a) would require some significant changes to
postgres to be performant, b) isn't really applicable for the large
percentage of installations that aren't tuned reasonably well, because
at the moment the OS page cache functions as a memory-pressure aware
extension of postgres' page cache.


Another topic brought up in this thread was the handling of ENOSPC
errors that aren't triggered on a filesystem level, but rather are
triggered by thin provisioning. On linux that currently apprently lead
to page cache contents being lost (and errors "eaten") in a lot of
places, including just when doing a write(). In a lot of cases it's
pretty much expected that the file system will just hang or react
unpredictably upon space exhaustion.  My reading is that the block-layer
thin provisioning code is still pretty fresh, and should only be used
with great care. The only way to halfway reliably use it appears to
change the configuration so space exhaustion blocks until admin
intervention (at least dm-thinp provides allows that).


There's some clear need to automate some more testing in this area so
that future behaviour changes don't surprise us.


== Proposed Linux Changes ==

- Matthew Wilcox proposed (and posted a patch) that'd partially revert
  behaviour to the pre v4.13 world, by *also* reporting errors to
  "newer" file-descriptors if the error hasn't previously been
  reported. That'd still not guarantee that the error is reported
  (memory pressure could evict information without open fd), but in most
  situations we'll again get the error in the checkpointer.

  This seems largely be agreed upon. It's unclear whether it'll go into
  the stable backports for still-maintained >= v4.13 kernels.

- syncfs() will be fixed so it reports errors properly - that'll likely
  require passing it an O_PATH filedescriptor to have space to store the
  errseq_t value that allows discerning already reported and new errors.

  No patch has appeared yet, but the behaviour seems largely agreed
  upon.

- Make per-filesystem error counts available in a uniform (i.e. same for
  every supporting fs) manner. Right now it's very hard to figure out
  whether errors occurred. There seemed general agreement that exporting
  knowledge about such errors is desirable. Quite possibly the syncfs()
  fix above will provide the necessary infrastructure. It's unclear as
  of yet how the value would be exposed. Per-fs /sys/ entries and an
  ioctl on O_PATH fds have been mentioned.

  These'd error counts would not vanish due to memory pressure, and they
  can be checked even without knowing which files in a specific
  filesystem have been touched (e.g. when just untar-ing something).

  There seemed to be fairly widespread agreement that this'd be a good
  idea. Much less clearer whether somebody would do the work.

- Provide config knobs that allow to define the FS error behaviour in a
  consistent way across supported filesystems. XFS currently has various
  knobs controlling what happens in case of metadata errors [1] (retry
  forever, timeout, return up). It was proposed that this interface be
  extended to also deal with data errors, and moved into generic support
  code.

  While the timeline is unclear, there seemed to be widespread support
  for the idea. I believe Dave Chinner indicated that he at least has
  plans to generalize the code.

- Stop inodes with unreported errors from being evicted. This will
  guarantee that a later fsync (without an open FD) will see the
  error. The memory pressure concerns here are lower than with keeping
  all the failed pages in memory, and it could be optimized further.

  I read some tentative agreement behind this idea, but I think it's the
  by far most controversial one.


== Potential Postgres Changes ==

Several operating systems / file systems behave differently (See
e.g. [2], thanks Thomas) than we expected. Even the discussed changes to
e.g. linux don't get to where we thought we are. There's obviously also
the question of how to deal with kernels / OSs that have not been
updated.

Changes that appear to be necessary, even for kernels with the issues
addressed:

- Clearly we need to treat fsync() EIO, ENOSPC errors as a PANIC and
  retry recovery. While ENODEV (underlying device went away) will be
  persistent, it probably makes sense to treat it the same or even just
  give up and shut down.  One question I see here is whether we just
  want to continue crash-recovery cycles, or whether we want to limit
  that.

- We need more aggressive error checking on close(), for ENOSPC and
  EIO. In both cases afaics we'll have to trigger a crash recovery
  cycle. It's entirely possible to end up in a loop on NFS etc, but I
  don't think there's a way around that.

  Robert, on IM, wondered whether there'd be a race between some backend
  doing a close(), triggering a PANIC, and a checkpoint succeeding.  I
  don't *think* so, because the error will only happen if there's
  outstanding dirty data, and the checkpoint would have flushed that out
  if it belonged to the current checkpointing cycle.

- The outstanding fsync request queue isn't persisted properly [3]. This
  means that even if the kernel behaved the way we'd expected, we'd not
  fail a second checkpoint :(. It's possible that we don't need to deal
  with this because we'll henceforth PANIC, but I'd argue we should fix
  that regardless. Seems like a time-bomb otherwise (e.g. after moving
  to DIO somebody might want to relax the PANIC...).

- It might be a good idea to whitelist expected return codes for write()
  and PANIC one ones that we did not expect. E.g. when hitting an EIO we
  should probably PANIC, to get back to a known good state. Even though
  it's likely that we'd again that error at fsync().

- Docs.

I think we also need to audit a few codepaths. I'd be surprised if we
PANICed appropriately on all fsyncs(), particularly around the SLRUs.  I
think we need to be particularly careful around the WAL handling, I
think it's fairly likely that there's cases where we'd write out WAL in
one backend and then fsync() in another backend with a file descriptor
that has only been opened *after* the write occurred, which means we
might miss the error entirely.


Then there's the question of how we want to deal with kernels that
haven't been updated with the aforementioned changes. We could say that
we expect decent OS support and declare that we just can't handle this -
given that at least various linux versions, netbsd, openbsd, MacOS just
silently drop errors and we'd need different approaches for dealing with
that, that doesn't seem like an insane approach.

What we could do:

- forward file descriptors from backends to checkpointer (using
  SCM_RIGHTS) when marking a segment dirty. That'd require some
  optimizations (see [4]) to avoid doing so repeatedly.  That'd
  guarantee correct behaviour in all linux kernels >= 4.13 (possibly
  backported by distributions?), and I think it'd also make it vastly
  more likely that errors are reported in earlier kernels.

  This should be doable without a noticeable performance impact, I
  believe.  I don't think it'd be that hard either, but it'd be a bit of
  a pain to backport it to all postgres versions, as well as a bit
  invasive for that.

  The infrastructure this'd likely end up building (hashtable of open
  relfilenodes), would likely be useful for further things (like caching
  file size).

- Add a pre-checkpoint hook that checks for filesystem errors *after*
  fsyncing all the files, but *before* logging the checkpoint completion
  record. Operating systems, filesystems, etc. all log the error format
  differently, but for larger installations it'd not be too hard to
  write code that checks their specific configuration.

  While I'm a bit concerned adding user-code before a checkpoint, if
  we'd do it as a shell command it seems pretty reasonable. And useful
  even without concern for the fsync issue itself. Checking for IO
  errors could e.g. also include checking for read errors - it'd not be
  unreasonable to not want to complete a checkpoint if there'd been any
  media errors.

- Use direct IO. Due to architectural performance issues in PG and the
  fact that it'd not be applicable for all installations I don't think
  this is a reasonable fix for the issue presented here. Although it's
  independently something we should work on.  It might be worthwhile to
  provide a configuration that allows to force DIO to be enabled for WAL
  even if replication is turned on.

- magic

Greetings,

Andres Freund

[0] https://archives.postgresql.org/message-id/CAMsr+YHh+5Oq4xziwwoEfhoTZgr07vdGG+hu=1adXx59aTeaoQ@mail.gmail.com

[1]
static const struct xfs_error_init xfs_error_meta_init[XFS_ERR_ERRNO_MAX] = {
    { .name = "default",
      .max_retries = XFS_ERR_RETRY_FOREVER,
      .retry_timeout = XFS_ERR_RETRY_FOREVER,
    },
    { .name = "EIO",
      .max_retries = XFS_ERR_RETRY_FOREVER,
      .retry_timeout = XFS_ERR_RETRY_FOREVER,
    },
    { .name = "ENOSPC",
      .max_retries = XFS_ERR_RETRY_FOREVER,
      .retry_timeout = XFS_ERR_RETRY_FOREVER,
    },
    { .name = "ENODEV",
      .max_retries = 0,    /* We can't recover from devices disappearing */
      .retry_timeout = 0,
    },
};

[2] https://wiki.postgresql.org/wiki/Fsync_Errors
[3] https://archives.postgresql.org/message-id/87y3i1ia4w.fsf%40news-spur.riddles.org.uk
[4] https://archives.postgresql.org/message-id/20180424180054.inih6bxfspgowjuc@alap3.anarazel.de


Re: Postgres, fsync, and OSs (specifically linux)

From
Bruce Momjian
Date:
On Fri, Apr 27, 2018 at 03:28:42PM -0700, Andres Freund wrote:
> - We need more aggressive error checking on close(), for ENOSPC and
>   EIO. In both cases afaics we'll have to trigger a crash recovery
>   cycle. It's entirely possible to end up in a loop on NFS etc, but I
>   don't think there's a way around that.

If the no-space or write failures are persistent, as you mentioned
above, what is the point of going into crash recovery --- why not just
shut down?  Also, since we can't guarantee that we can write any
persistent state to storage, we have no way of preventing infinite crash
recovery loops, which, based on inconsistent writes, might make things
worse.  I think a single panic with no restart is the right solution.

An additional features we have talked about is running some kind of
notification shell script to inform administrators, similar to
archive_command.  We need this too when sync replication fails.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +


Re: Postgres, fsync, and OSs (specifically linux)

From
Andres Freund
Date:
Hi,

On 2018-04-27 19:04:47 -0400, Bruce Momjian wrote:
> On Fri, Apr 27, 2018 at 03:28:42PM -0700, Andres Freund wrote:
> > - We need more aggressive error checking on close(), for ENOSPC and
> >   EIO. In both cases afaics we'll have to trigger a crash recovery
> >   cycle. It's entirely possible to end up in a loop on NFS etc, but I
> >   don't think there's a way around that.
> 
> If the no-space or write failures are persistent, as you mentioned
> above, what is the point of going into crash recovery --- why not just
> shut down?

Well, I mentioned that as an alternative in my email. But for one we
don't really have cases where we do that right now, for another we can't
really differentiate between a transient and non-transient state. It's
entirely possible that the admin on the system that ran out of space
fixes things, clearing up the problem.


> Also, since we can't guarantee that we can write any persistent state
> to storage, we have no way of preventing infinite crash recovery
> loops, which, based on inconsistent writes, might make things worse.

How would it make things worse?


> An additional features we have talked about is running some kind of
> notification shell script to inform administrators, similar to
> archive_command.  We need this too when sync replication fails.

To me that seems like a feature independent of this thread.

Greetings,

Andres Freund


Re: Postgres, fsync, and OSs (specifically linux)

From
Bruce Momjian
Date:
On Fri, Apr 27, 2018 at 04:10:43PM -0700, Andres Freund wrote:
> Hi,
> 
> On 2018-04-27 19:04:47 -0400, Bruce Momjian wrote:
> > On Fri, Apr 27, 2018 at 03:28:42PM -0700, Andres Freund wrote:
> > > - We need more aggressive error checking on close(), for ENOSPC and
> > >   EIO. In both cases afaics we'll have to trigger a crash recovery
> > >   cycle. It's entirely possible to end up in a loop on NFS etc, but I
> > >   don't think there's a way around that.
> > 
> > If the no-space or write failures are persistent, as you mentioned
> > above, what is the point of going into crash recovery --- why not just
> > shut down?
> 
> Well, I mentioned that as an alternative in my email. But for one we
> don't really have cases where we do that right now, for another we can't
> really differentiate between a transient and non-transient state. It's
> entirely possible that the admin on the system that ran out of space
> fixes things, clearing up the problem.

True, but if we get a no-space error, odds are it will not be fixed at
the time we are failing.  Wouldn't the administrator check that the
server is still running after they free the space?

> > Also, since we can't guarantee that we can write any persistent state
> > to storage, we have no way of preventing infinite crash recovery
> > loops, which, based on inconsistent writes, might make things worse.
> 
> How would it make things worse?

Uh, I can imagine some writes working and some not, and getting things
more inconsistent.  I would say at least that we don't know.

> > An additional features we have talked about is running some kind of
> > notification shell script to inform administrators, similar to
> > archive_command.  We need this too when sync replication fails.
> 
> To me that seems like a feature independent of this thread.

Well, if we are introducing new panic-and-not-restart behavior, we might
need this new feature.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +


Re: Postgres, fsync, and OSs (specifically linux)

From
Andres Freund
Date:
On 2018-04-27 19:38:30 -0400, Bruce Momjian wrote:
> On Fri, Apr 27, 2018 at 04:10:43PM -0700, Andres Freund wrote:
> > Hi,
> > 
> > On 2018-04-27 19:04:47 -0400, Bruce Momjian wrote:
> > > On Fri, Apr 27, 2018 at 03:28:42PM -0700, Andres Freund wrote:
> > > > - We need more aggressive error checking on close(), for ENOSPC and
> > > >   EIO. In both cases afaics we'll have to trigger a crash recovery
> > > >   cycle. It's entirely possible to end up in a loop on NFS etc, but I
> > > >   don't think there's a way around that.
> > > 
> > > If the no-space or write failures are persistent, as you mentioned
> > > above, what is the point of going into crash recovery --- why not just
> > > shut down?
> > 
> > Well, I mentioned that as an alternative in my email. But for one we
> > don't really have cases where we do that right now, for another we can't
> > really differentiate between a transient and non-transient state. It's
> > entirely possible that the admin on the system that ran out of space
> > fixes things, clearing up the problem.
> 
> True, but if we get a no-space error, odds are it will not be fixed at
> the time we are failing.  Wouldn't the administrator check that the
> server is still running after they free the space?

I'd assume it's pretty common that those are separate teams. Given that
we currently don't behave that way for other cases where we *already*
can enter crash-recovery loops I don't think we need to introduce that
here. It's far more common to enter this kind of problem with pg_xlog
filling up the ordinary way. And that can lead to such loops.


> > > Also, since we can't guarantee that we can write any persistent state
> > > to storage, we have no way of preventing infinite crash recovery
> > > loops, which, based on inconsistent writes, might make things worse.
> > 
> > How would it make things worse?
> 
> Uh, I can imagine some writes working and some not, and getting things
> more inconsistent.  I would say at least that we don't know.

Recovery needs to fix that or we're lost anyway. And we'll retry exactly
the same writes each round.


> > > An additional features we have talked about is running some kind of
> > > notification shell script to inform administrators, similar to
> > > archive_command.  We need this too when sync replication fails.
> > 
> > To me that seems like a feature independent of this thread.
> 
> Well, if we are introducing new panic-and-not-restart behavior, we might
> need this new feature.

I don't see how this follows. It's easier to externally script
notification for the server having died, than doing it for crash
restarts. That's why we have restart_after_crash=false...  There might
be some arguments for this type of notification, but I don't think it
should be conflated with the problem here. Nor is it guaranteed that
such a script could do much, given that disks might be failing and such.

Greetings,

Andres Freund


Re: Postgres, fsync, and OSs (specifically linux)

From
Craig Ringer
Date:
On 28 April 2018 at 06:28, Andres Freund <andres@anarazel.de> wrote:
> Hi,
>
> I thought I'd send this separately from [0] as the issue has become more
> general than what was mentioned in that thread, and it went off into
> various weeds.

Thanks very much for going and for the great summary.

> - Actually marking files as persistently failed would require filesystem
>   changes, and filesystem metadata IO, far from guaranteed in failure
>   scenarios.

Yeah, I've avoided suggesting anything like that because it seems way
too likely to lead to knock-on errors.

Like malloc'ing in an OOM path, just don't.

> The second major type of proposal was using direct-IO. That'd generally
> be a desirable feature, but a) would require some significant changes to
> postgres to be performant, b) isn't really applicable for the large
> percentage of installations that aren't tuned reasonably well, because
> at the moment the OS page cache functions as a memory-pressure aware
> extension of postgres' page cache.

Yeah. I've avoided advocating for O_DIRECT because it's a big job
(understatement). We'd need to pay so much more attention to details
of storage layout if we couldn't rely as much on the kernel neatly
organising and queuing everything for us, too.

At the risk of displaying my relative ignorance of direct I/O: Does
O_DIRECT without O_SYNC even provide a strong guarantee that when you
close() the file, all I/O has reliably succeeded? It must've gone
through the FS layer, but don't FSes do various caching and
reorganisation too? Can the same issue arise in other ways unless we
also fsync() before close() or write O_SYNC?

At one point I looked into using AIO instead. But last I looked it was
pretty spectacularly quirky when it comes to reliably flushing, and
outright broken on some versions. In any case, our multiprocessing
model would make tracking completions annoying, likely more so than
the sort of FD handoff games we've discussed.

> Another topic brought up in this thread was the handling of ENOSPC
> errors that aren't triggered on a filesystem level, but rather are
> triggered by thin provisioning. On linux that currently apprently lead
> to page cache contents being lost (and errors "eaten") in a lot of
> places, including just when doing a write().

... wow.

Is that with lvm-thin?

The thin provisioning I was mainly concerned with is SAN-based thin
provisioning, which looks like a normal iSCSI target or a normal LUN
on a HBA to Linux. Then it starts failing writes with a weird
potentially vendor-specific sense error if it runs out of backing
store. How that's handled likely depends on the specific error, the
driver, which FS you use, etc. In the case I saw, multipath+lvm+xfs,
it resulted in lost writes and fsync() errors being reported once, per
the start of the original thread.

> In a lot of cases it's
> pretty much expected that the file system will just hang or react
> unpredictably upon space exhaustion.  My reading is that the block-layer
> thin provisioning code is still pretty fresh, and should only be used
> with great care. The only way to halfway reliably use it appears to
> change the configuration so space exhaustion blocks until admin
> intervention (at least dm-thinp provides allows that).

Seems that should go in the OS-specific configuration part of the
docs, along with the advice I gave on the original thread re
configuring multipath no_path_retries.

> There's some clear need to automate some more testing in this area so
> that future behaviour changes don't surprise us.

We don't routinely test ENOSPC (or memory exhaustion, or crashes) in
PostgreSQL even on bog standard setups.

Like the performance farm discussion, this is something I'd like to
pick up at some point. I'm going to need to talk to the team I work
with regarding time/resources allocation, but I think it's important
that we make such testing more of a routine thing.

> - Matthew Wilcox proposed (and posted a patch) that'd partially revert
>   behaviour to the pre v4.13 world, by *also* reporting errors to
>   "newer" file-descriptors if the error hasn't previously been
>   reported. That'd still not guarantee that the error is reported
>   (memory pressure could evict information without open fd), but in most
>   situations we'll again get the error in the checkpointer.
>
>   This seems largely be agreed upon. It's unclear whether it'll go into
>   the stable backports for still-maintained >= v4.13 kernels.

That seems very sensible. In our case we're very unlikely to have some
other unrelated process come in and fsync() our files for us.

I'd want to be sure the report didn't get eaten by sync() or syncfs() though.

> - syncfs() will be fixed so it reports errors properly - that'll likely
>   require passing it an O_PATH filedescriptor to have space to store the
>   errseq_t value that allows discerning already reported and new errors.
>
>   No patch has appeared yet, but the behaviour seems largely agreed
>   upon.

Good, but as you noted, of limited use to us unless we want to force
users to manage space for temporary and unlogged relations completely
separately.

I wonder if we could convince the kernel to offer a file_sync_mode
xattr to control this? (Hint: I'm already running away in a mylar fire
suit).

> - Make per-filesystem error counts available in a uniform (i.e. same for
>   every supporting fs) manner. Right now it's very hard to figure out
>   whether errors occurred. There seemed general agreement that exporting
>   knowledge about such errors is desirable. Quite possibly the syncfs()
>   fix above will provide the necessary infrastructure. It's unclear as
>   of yet how the value would be exposed. Per-fs /sys/ entries and an
>   ioctl on O_PATH fds have been mentioned.
>
>   These'd error counts would not vanish due to memory pressure, and they
>   can be checked even without knowing which files in a specific
>   filesystem have been touched (e.g. when just untar-ing something).
>
>   There seemed to be fairly widespread agreement that this'd be a good
>   idea. Much less clearer whether somebody would do the work.
>
> - Provide config knobs that allow to define the FS error behaviour in a
>   consistent way across supported filesystems. XFS currently has various
>   knobs controlling what happens in case of metadata errors [1] (retry
>   forever, timeout, return up). It was proposed that this interface be
>   extended to also deal with data errors, and moved into generic support
>   code.
>
>   While the timeline is unclear, there seemed to be widespread support
>   for the idea. I believe Dave Chinner indicated that he at least has
>   plans to generalize the code.

That's great. It sounds like this has revitalised some interest in the
error reporting and might yield some more general cleanups :)

> - Stop inodes with unreported errors from being evicted. This will
>   guarantee that a later fsync (without an open FD) will see the
>   error. The memory pressure concerns here are lower than with keeping
>   all the failed pages in memory, and it could be optimized further.
>
>   I read some tentative agreement behind this idea, but I think it's the
>   by far most controversial one.

The main issue there would seem to be cases of whole-FS failure like
the USB-key-yank example. You're going to have to be able to get rid
of them at some point.

> - Clearly we need to treat fsync() EIO, ENOSPC errors as a PANIC and
>   retry recovery. While ENODEV (underlying device went away) will be
>   persistent, it probably makes sense to treat it the same or even just
>   give up and shut down.  One question I see here is whether we just
>   want to continue crash-recovery cycles, or whether we want to limit
>   that.

Right now, we'll panic once, then panic again in redo if the error
persists and give up.

On some systems, and everywhere that Pg is directly user-managed with
pg_ctl, that'll leave Pg down until the operator intervenes. Some init
systems will restart the postmaster automatically. Some will give up
after a few tries. Some will back off retries over time. It depends on
the init system. I'm not sure that's a great outcome.

So rather than giving up if redo fails, we might want to offer a knob
to retry, possibly with pause/backoff. I'm sure people currently
expect PostgreSQL to try to stay up and recover, like it does after a
segfault or most other errors.

Personally I prefer to run Pg with restart_after_crash=off and let the
init system launch a new postmaster, but that's not an option unless
you have a sensible init.

> - We need more aggressive error checking on close(), for ENOSPC and
>   EIO. In both cases afaics we'll have to trigger a crash recovery
>   cycle. It's entirely possible to end up in a loop on NFS etc, but I
>   don't think there's a way around that.
>
>   Robert, on IM, wondered whether there'd be a race between some backend
>   doing a close(), triggering a PANIC, and a checkpoint succeeding.  I
>   don't *think* so, because the error will only happen if there's
>   outstanding dirty data, and the checkpoint would have flushed that out
>   if it belonged to the current checkpointing cycle.

Even if it's possible (which it sounds like it probably isn't), it
might also be one of those corner-cases-of-corner-cases where we just
shrug and worry about bigger fish.

> - The outstanding fsync request queue isn't persisted properly [3]. This
>   means that even if the kernel behaved the way we'd expected, we'd not
>   fail a second checkpoint :(. It's possible that we don't need to deal
>   with this because we'll henceforth PANIC, but I'd argue we should fix
>   that regardless. Seems like a time-bomb otherwise (e.g. after moving
>   to DIO somebody might want to relax the PANIC...).

Huh! Good find. That definitely merits fixing.

> - It might be a good idea to whitelist expected return codes for write()
>   and PANIC one ones that we did not expect. E.g. when hitting an EIO we
>   should probably PANIC, to get back to a known good state. Even though
>   it's likely that we'd again that error at fsync().
>
> - Docs.

Yep. Especially OS-specific configuration for known dangerous setups
(lvm-thin, multipath), etc. I imagine we can distill a lot of it from
the discussion and simplify a bit.

> I think we also need to audit a few codepaths. I'd be surprised if we
> PANICed appropriately on all fsyncs(), particularly around the SLRUs.

We _definitely_ do not, see the patch I sent on the other thread.

> Then there's the question of how we want to deal with kernels that
> haven't been updated with the aforementioned changes. We could say that
> we expect decent OS support and declare that we just can't handle this -
> given that at least various linux versions, netbsd, openbsd, MacOS just
> silently drop errors and we'd need different approaches for dealing with
> that, that doesn't seem like an insane approach.

> What we could do:
>
> - forward file descriptors from backends to checkpointer (using
>   SCM_RIGHTS) when marking a segment dirty. That'd require some
>   optimizations (see [4]) to avoid doing so repeatedly.  That'd
>   guarantee correct behaviour in all linux kernels >= 4.13 (possibly
>   backported by distributions?), and I think it'd also make it vastly
>   more likely that errors are reported in earlier kernels.

It'd be interesting to see if other platforms that support fd passing
will give us the desired behaviour too. But even if it only helps on
Linux, that's a huge majority of the PostgreSQL deployments these
days.

> - Add a pre-checkpoint hook that checks for filesystem errors *after*
>   fsyncing all the files, but *before* logging the checkpoint completion
>   record. Operating systems, filesystems, etc. all log the error format
>   differently, but for larger installations it'd not be too hard to
>   write code that checks their specific configuration.

I looked into using trace event file descriptors for this, btw, but
we'd need CAP_SYS_ADMIN to create one that captured events for other
processes. Plus filtering the events to find only events for the files
/ file systems of interest would be far from trivial. And I don't know
what guarantees we have about when events are delivered.

I'd love to be able to use inotify for this, but again, that'd only be
a new-kernels thing since it'd need an inotify extension to report I/O
errors.

Presumably mostly this check would land up looking at dmesg.

I'm not convinced it'd get widely deployed and widely used, or that
it'd be used correctly when people tried to use it. Look at the
hideous mess that most backup/standby creation scripts,
archive_command scripts, etc are.

> - Use direct IO. Due to architectural performance issues in PG and the
>   fact that it'd not be applicable for all installations I don't think
>   this is a reasonable fix for the issue presented here. Although it's
>   independently something we should work on.  It might be worthwhile to
>   provide a configuration that allows to force DIO to be enabled for WAL
>   even if replication is turned on.

Seems like a long term goal, but you've noted elsewhere that doing it
well would be hard. I suspect we'd need writer threads, we'd need to
know more about the underlying FS/storage layout to make better
decisions about write parallelism, etc. We get away with a lot right
now by letting the kernel and buffered I/O sort that out.

-- 
 Craig Ringer                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services


Re: Postgres, fsync, and OSs (specifically linux)

From
Stephen Frost
Date:
Greetings,

* Andres Freund (andres@anarazel.de) wrote:
> On 2018-04-27 19:38:30 -0400, Bruce Momjian wrote:
> > On Fri, Apr 27, 2018 at 04:10:43PM -0700, Andres Freund wrote:
> > > On 2018-04-27 19:04:47 -0400, Bruce Momjian wrote:
> > > > On Fri, Apr 27, 2018 at 03:28:42PM -0700, Andres Freund wrote:
> > > > > - We need more aggressive error checking on close(), for ENOSPC and
> > > > >   EIO. In both cases afaics we'll have to trigger a crash recovery
> > > > >   cycle. It's entirely possible to end up in a loop on NFS etc, but I
> > > > >   don't think there's a way around that.
> > > >
> > > > If the no-space or write failures are persistent, as you mentioned
> > > > above, what is the point of going into crash recovery --- why not just
> > > > shut down?
> > >
> > > Well, I mentioned that as an alternative in my email. But for one we
> > > don't really have cases where we do that right now, for another we can't
> > > really differentiate between a transient and non-transient state. It's
> > > entirely possible that the admin on the system that ran out of space
> > > fixes things, clearing up the problem.
> >
> > True, but if we get a no-space error, odds are it will not be fixed at
> > the time we are failing.  Wouldn't the administrator check that the
> > server is still running after they free the space?
>
> I'd assume it's pretty common that those are separate teams. Given that
> we currently don't behave that way for other cases where we *already*
> can enter crash-recovery loops I don't think we need to introduce that
> here. It's far more common to enter this kind of problem with pg_xlog
> filling up the ordinary way. And that can lead to such loops.

When we crash-restart, we also go through and clean things up some, no?
Seems like that gives us the potential to end up fixing things ourselves
and allowing the crash-restart to succeed.

Consider unlogged tables, temporary tables, on-disk sorts, etc.  It's
entirely common for a bad query to run the system out of disk space (but
have a write of a regular table be what discovers the out-of-space
problem...) and if we crash-restart properly then we'd hopefully clean
things out, freeing up space, and allowing us to come back up.

Now, of course, ideally admins would set up temp tablespaces and
segregate WAL onto its own filesystem, etc, but...

Thanks!

Stephen

Attachment

Re: Postgres, fsync, and OSs (specifically linux)

From
Stephen Frost
Date:
Greetings,

* Craig Ringer (craig@2ndquadrant.com) wrote:
> On 28 April 2018 at 06:28, Andres Freund <andres@anarazel.de> wrote:
> > - Add a pre-checkpoint hook that checks for filesystem errors *after*
> >   fsyncing all the files, but *before* logging the checkpoint completion
> >   record. Operating systems, filesystems, etc. all log the error format
> >   differently, but for larger installations it'd not be too hard to
> >   write code that checks their specific configuration.
>
> I looked into using trace event file descriptors for this, btw, but
> we'd need CAP_SYS_ADMIN to create one that captured events for other
> processes. Plus filtering the events to find only events for the files
> / file systems of interest would be far from trivial. And I don't know
> what guarantees we have about when events are delivered.
>
> I'd love to be able to use inotify for this, but again, that'd only be
> a new-kernels thing since it'd need an inotify extension to report I/O
> errors.
>
> Presumably mostly this check would land up looking at dmesg.
>
> I'm not convinced it'd get widely deployed and widely used, or that
> it'd be used correctly when people tried to use it. Look at the
> hideous mess that most backup/standby creation scripts,
> archive_command scripts, etc are.

Agree with more-or-less everything you've said here, but a big +1 on
this.  If we do end up going down this route we have *got* to provide
scripts which we know work and have been tested and are well maintained
on the popular OS's for the popular filesystems and make it clear that
we've tested those and not others.  We definitely shouldn't put
something in our docs that is effectively an example of the interface
but not an actual command that anyone should be using.

Thanks!

Stephen

Attachment

Re: Postgres, fsync, and OSs (specifically linux)

From
Simon Riggs
Date:
On 27 April 2018 at 15:28, Andres Freund <andres@anarazel.de> wrote:

> - Add a pre-checkpoint hook that checks for filesystem errors *after*
>   fsyncing all the files, but *before* logging the checkpoint completion
>   record. Operating systems, filesystems, etc. all log the error format
>   differently, but for larger installations it'd not be too hard to
>   write code that checks their specific configuration.
>
>   While I'm a bit concerned adding user-code before a checkpoint, if
>   we'd do it as a shell command it seems pretty reasonable. And useful
>   even without concern for the fsync issue itself. Checking for IO
>   errors could e.g. also include checking for read errors - it'd not be
>   unreasonable to not want to complete a checkpoint if there'd been any
>   media errors.

It seems clear that we need to evaluate our compatibility not just
with an OS, as we do now, but with an OS/filesystem.

Although people have suggested some approaches, I'm more interested in
discovering how we can be certain we got it right.

And the end result seems to be that PostgreSQL will be forced, in the
short term, to declare certain combinations of OS/filesystem
unsupported, with clear warning sent out to users.

Adding a pre-checkpoint hook encourages people to fix this themselves
without reporting issues, so I initially oppose this until we have a
clearer argument as to why we need it. The answer is not to make this
issue more obscure, but to make it more public.

> - Use direct IO. Due to architectural performance issues in PG and the
>   fact that it'd not be applicable for all installations I don't think
>   this is a reasonable fix for the issue presented here. Although it's
>   independently something we should work on.  It might be worthwhile to
>   provide a configuration that allows to force DIO to be enabled for WAL
>   even if replication is turned on.

"Use DirectIO" is roughly same suggestion as "don't trust Linux filesystems".

It would be a major admission of defeat for us to take that as our
main route to a solution.

The people I've spoken to so far have encouraged us to continue
working with the filesystem layer, offering encouragement of our
decision to use filesystems.

-- 
Simon Riggs                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services


Re: Postgres, fsync, and OSs (specifically linux)

From
Michael Banck
Date:
Hi,

On Sat, Apr 28, 2018 at 11:21:20AM -0400, Stephen Frost wrote:
> * Craig Ringer (craig@2ndquadrant.com) wrote:
> > On 28 April 2018 at 06:28, Andres Freund <andres@anarazel.de> wrote:
> > > - Add a pre-checkpoint hook that checks for filesystem errors *after*
> > >   fsyncing all the files, but *before* logging the checkpoint completion
> > >   record. Operating systems, filesystems, etc. all log the error format
> > >   differently, but for larger installations it'd not be too hard to
> > >   write code that checks their specific configuration.
> > 
> > I looked into using trace event file descriptors for this, btw, but
> > we'd need CAP_SYS_ADMIN to create one that captured events for other
> > processes. Plus filtering the events to find only events for the files
> > / file systems of interest would be far from trivial. And I don't know
> > what guarantees we have about when events are delivered.
> > 
> > I'd love to be able to use inotify for this, but again, that'd only be
> > a new-kernels thing since it'd need an inotify extension to report I/O
> > errors.
> > 
> > Presumably mostly this check would land up looking at dmesg.
> > 
> > I'm not convinced it'd get widely deployed and widely used, or that
> > it'd be used correctly when people tried to use it. Look at the
> > hideous mess that most backup/standby creation scripts,
> > archive_command scripts, etc are.
> 
> Agree with more-or-less everything you've said here, but a big +1 on
> this.  If we do end up going down this route we have *got* to provide
> scripts which we know work and have been tested and are well maintained
> on the popular OS's for the popular filesystems and make it clear that
> we've tested those and not others.  We definitely shouldn't put
> something in our docs that is effectively an example of the interface
> but not an actual command that anyone should be using.

This dmesg-checking has been mentioned several times now, but IME
enterprise distributions (or server ops teams?) seem to tighten access
to dmesg and /var/log to non-root users, including postgres.

Well, or just vanilla Debian stable apparently:

postgres@fock:~$ dmesg
dmesg: read kernel buffer failed: Operation not permitted

Is it really a useful expectation that the postgres user will be able to
trawl system logs for I/O errors? Or are we expecting the sysadmins (in
case they are distinct from the DBAs) to setup sudo and/or relax
permissions for this everywhere? We should document this requirement
properly at least then.

The netlink thing from Google that Tet Ts'O mentioned would probably
work around that, but if that is opened up it would not be deployed
anytime soon either.


Michael

-- 
Michael Banck
Projektleiter / Senior Berater
Tel.: +49 2166 9901-171
Fax:  +49 2166 9901-100
Email: michael.banck@credativ.de 
credativ GmbH, HRB Mönchengladbach 12080
USt-ID-Nummer: DE204566209
Trompeterallee 108, 41189 Mönchengladbach
Geschäftsführung: Dr. Michael Meskes, Jörg Folz, Sascha Heuer


Re: Postgres, fsync, and OSs (specifically linux)

From
Andres Freund
Date:
Hi,

On 2018-04-28 11:10:54 -0400, Stephen Frost wrote:
> When we crash-restart, we also go through and clean things up some, no?
> Seems like that gives us the potential to end up fixing things ourselves
> and allowing the crash-restart to succeed.

Sure, there's the potential for that. But it's quite possible to be
missing a lot of free space over NFS (this really isn't much of an issue
for local FS, at least not on linux) in a workload with rapidly
expanding space usage. And even if you recover, you could just hit the
issue again shortly afterwards.

Greetings,

Andres Freund


Re: Postgres, fsync, and OSs (specifically linux)

From
Andres Freund
Date:
Hi,

On 2018-04-28 17:35:48 +0200, Michael Banck wrote:
> This dmesg-checking has been mentioned several times now, but IME
> enterprise distributions (or server ops teams?) seem to tighten access
> to dmesg and /var/log to non-root users, including postgres.
> 
> Well, or just vanilla Debian stable apparently:
> 
> postgres@fock:~$ dmesg
> dmesg: read kernel buffer failed: Operation not permitted
> 
> Is it really a useful expectation that the postgres user will be able to
> trawl system logs for I/O errors? Or are we expecting the sysadmins (in
> case they are distinct from the DBAs) to setup sudo and/or relax
> permissions for this everywhere? We should document this requirement
> properly at least then.

I'm not a huge fan of this approach, but yes, that'd be necessary. It's
not that problematic to have to change /dev/kmsg permissions imo. Adding
a read group / acl seems quite doable.


> The netlink thing from Google that Tet Ts'O mentioned would probably
> work around that, but if that is opened up it would not be deployed
> anytime soon either.

Yea, that seems irrelevant for now.

Greetings,

Andres Freund


Re: Postgres, fsync, and OSs (specifically linux)

From
Andres Freund
Date:
Hi,

On 2018-04-28 08:25:53 -0700, Simon Riggs wrote:
> > - Use direct IO. Due to architectural performance issues in PG and the
> >   fact that it'd not be applicable for all installations I don't think
> >   this is a reasonable fix for the issue presented here. Although it's
> >   independently something we should work on.  It might be worthwhile to
> >   provide a configuration that allows to force DIO to be enabled for WAL
> >   even if replication is turned on.
> 
> "Use DirectIO" is roughly same suggestion as "don't trust Linux filesystems".

I want to emphasize that this is NOT a linux only issue. It's a problem
across a number of operating systems, including linux.


> It would be a major admission of defeat for us to take that as our
> main route to a solution.

Well, I think we were wrong to not engineer towards DIO. There's just
too many issues with buffered IO to not have a supported path for
DIO. But given that it's unrealistic to do so without major work, and
wouldn't be applicable for all installations (shared_buffer size becomes
critical), I don't think it matters that much for the issue discussed
here.


> The people I've spoken to so far have encouraged us to continue
> working with the filesystem layer, offering encouragement of our
> decision to use filesystems.

There's a lot of people disagreeing with it too.

Greetings,

Andres Freund


Re: Postgres, fsync, and OSs (specifically linux)

From
Andres Freund
Date:
Hi,

On 2018-04-28 20:00:25 +0800, Craig Ringer wrote:
> On 28 April 2018 at 06:28, Andres Freund <andres@anarazel.de> wrote:
> > The second major type of proposal was using direct-IO. That'd generally
> > be a desirable feature, but a) would require some significant changes to
> > postgres to be performant, b) isn't really applicable for the large
> > percentage of installations that aren't tuned reasonably well, because
> > at the moment the OS page cache functions as a memory-pressure aware
> > extension of postgres' page cache.
> 
> Yeah. I've avoided advocating for O_DIRECT because it's a big job
> (understatement). We'd need to pay so much more attention to details
> of storage layout if we couldn't rely as much on the kernel neatly
> organising and queuing everything for us, too.
> 
> At the risk of displaying my relative ignorance of direct I/O: Does
> O_DIRECT without O_SYNC even provide a strong guarantee that when you
> close() the file, all I/O has reliably succeeded? It must've gone
> through the FS layer, but don't FSes do various caching and
> reorganisation too? Can the same issue arise in other ways unless we
> also fsync() before close() or write O_SYNC?

No, not really. There's generally two categories of IO here: Metadata IO
and data IO. The filesystem's metadata IO a) has a lot more error
checking (including things like remount-ro, stalling the filesystem on
errors etc), b) isn't direct IO itself.  For some filesystem metadata
operations you'll still need fsyncs, but the *data* is flushed if use
use DIO. I'd personally use O_DSYNC | O_DIRECT, and have the metadata
operations guaranteed by fsyncs.  You'd need the current fsyncs for
renaming, and probably some fsyncs for file extensions. The latter to
make sure the filesystem has written the metadata change.


> At one point I looked into using AIO instead. But last I looked it was
> pretty spectacularly quirky when it comes to reliably flushing, and
> outright broken on some versions. In any case, our multiprocessing
> model would make tracking completions annoying, likely more so than
> the sort of FD handoff games we've discussed.

AIO pretty much only works sensibly with DIO.


> > Another topic brought up in this thread was the handling of ENOSPC
> > errors that aren't triggered on a filesystem level, but rather are
> > triggered by thin provisioning. On linux that currently apprently lead
> > to page cache contents being lost (and errors "eaten") in a lot of
> > places, including just when doing a write().
> 
> ... wow.
> 
> Is that with lvm-thin?

I think both dm and lvm (I typed llvm thrice) based thin
provisioning. The FS code basically didn't expect ENOSPC being returned
from storage, but suddenly the storage layer started returning it...


> The thin provisioning I was mainly concerned with is SAN-based thin
> provisioning, which looks like a normal iSCSI target or a normal LUN
> on a HBA to Linux. Then it starts failing writes with a weird
> potentially vendor-specific sense error if it runs out of backing
> store. How that's handled likely depends on the specific error, the
> driver, which FS you use, etc. In the case I saw, multipath+lvm+xfs,
> it resulted in lost writes and fsync() errors being reported once, per
> the start of the original thread.

I think the concerns are largely the same for that. You'll have to
configure the SAN to block in that case.


> > - Matthew Wilcox proposed (and posted a patch) that'd partially revert
> >   behaviour to the pre v4.13 world, by *also* reporting errors to
> >   "newer" file-descriptors if the error hasn't previously been
> >   reported. That'd still not guarantee that the error is reported
> >   (memory pressure could evict information without open fd), but in most
> >   situations we'll again get the error in the checkpointer.
> >
> >   This seems largely be agreed upon. It's unclear whether it'll go into
> >   the stable backports for still-maintained >= v4.13 kernels.
> 
> That seems very sensible. In our case we're very unlikely to have some
> other unrelated process come in and fsync() our files for us.
> 
> I'd want to be sure the report didn't get eaten by sync() or syncfs() though.

It doesn't. Basically every fd has an errseq_t value copied into it at
open.


> > - syncfs() will be fixed so it reports errors properly - that'll likely
> >   require passing it an O_PATH filedescriptor to have space to store the
> >   errseq_t value that allows discerning already reported and new errors.
> >
> >   No patch has appeared yet, but the behaviour seems largely agreed
> >   upon.
> 
> Good, but as you noted, of limited use to us unless we want to force
> users to manage space for temporary and unlogged relations completely
> separately.

Well, I think it'd still be ok as a backstop if it had decent error
semantics. We don't checkpoint that often, and doing the syncing via
syncfs() is considerably more efficient than individual fsync()s. But
given it's currently buggy that tradeoff is moot.


> I wonder if we could convince the kernel to offer a file_sync_mode
> xattr to control this? (Hint: I'm already running away in a mylar fire
> suit).

Err. I am fairly sure you're not going to get anywhere with that. Given
we're concerned about existing kernels, I doubt it'd help us much anyway.


> > - Stop inodes with unreported errors from being evicted. This will
> >   guarantee that a later fsync (without an open FD) will see the
> >   error. The memory pressure concerns here are lower than with keeping
> >   all the failed pages in memory, and it could be optimized further.
> >
> >   I read some tentative agreement behind this idea, but I think it's the
> >   by far most controversial one.
> 
> The main issue there would seem to be cases of whole-FS failure like
> the USB-key-yank example. You're going to have to be able to get rid
> of them at some point.

It's not actually a real problem (despite initially being brought up a
number of times by kernel people). There's a separate error for that
(ENODEV), and filesystems already handle it differently. Once that's
returned, fsyncs() etc are just shortcut to ENODEV.


> > What we could do:
> >
> > - forward file descriptors from backends to checkpointer (using
> >   SCM_RIGHTS) when marking a segment dirty. That'd require some
> >   optimizations (see [4]) to avoid doing so repeatedly.  That'd
> >   guarantee correct behaviour in all linux kernels >= 4.13 (possibly
> >   backported by distributions?), and I think it'd also make it vastly
> >   more likely that errors are reported in earlier kernels.
> 
> It'd be interesting to see if other platforms that support fd passing
> will give us the desired behaviour too. But even if it only helps on
> Linux, that's a huge majority of the PostgreSQL deployments these
> days.

Afaict it'd not help all of them. It does provide guarantees against the
inode being evicted on pretty much all OSs, but not all of them have an
error counter there...


> > - Use direct IO. Due to architectural performance issues in PG and the
> >   fact that it'd not be applicable for all installations I don't think
> >   this is a reasonable fix for the issue presented here. Although it's
> >   independently something we should work on.  It might be worthwhile to
> >   provide a configuration that allows to force DIO to be enabled for WAL
> >   even if replication is turned on.
> 
> Seems like a long term goal, but you've noted elsewhere that doing it
> well would be hard. I suspect we'd need writer threads, we'd need to
> know more about the underlying FS/storage layout to make better
> decisions about write parallelism, etc. We get away with a lot right
> now by letting the kernel and buffered I/O sort that out.

We're a *lot* slower due to it.

Don't think you would need writer threads, "just" a bgwriter that
actually works and provides clean buffers unless the machine is
overloaded. I've posted a patch that adds that. On the write side you
then additionally need write combining (doing one writes for several
on-disk-consecutive buffers), which isn't trivial to add currently.  The
bigger issue than writes is actually doing reads nicely. There's no
readahead anymore, and we'd not have the kernel backstopping our bad
caching decisions anymore.

Greetings,

Andres Freund


Re: Postgres, fsync, and OSs (specifically linux)

From
Craig Ringer
Date:
On 29 April 2018 at 00:15, Andres Freund <andres@anarazel.de> wrote:
> Hi,
>
> On 2018-04-28 08:25:53 -0700, Simon Riggs wrote:
>> > - Use direct IO. Due to architectural performance issues in PG and the
>> >   fact that it'd not be applicable for all installations I don't think
>> >   this is a reasonable fix for the issue presented here. Although it's
>> >   independently something we should work on.  It might be worthwhile to
>> >   provide a configuration that allows to force DIO to be enabled for WAL
>> >   even if replication is turned on.
>>
>> "Use DirectIO" is roughly same suggestion as "don't trust Linux filesystems".
>
> I want to emphasize that this is NOT a linux only issue. It's a problem
> across a number of operating systems, including linux.
>
>
>> It would be a major admission of defeat for us to take that as our
>> main route to a solution.
>
> Well, I think we were wrong to not engineer towards DIO. There's just
> too many issues with buffered IO to not have a supported path for
> DIO. But given that it's unrealistic to do so without major work, and
> wouldn't be applicable for all installations (shared_buffer size becomes
> critical), I don't think it matters that much for the issue discussed
> here.


20/20 hindsight, really. Not much to be done now.

Even with the work you and others have done on shared_buffers
scalability, there's likely still improvement needed there if it
becomes more important to evict buffers into per-device queues, etc,
too.

Personally I'd rather not have to write half the kernel's job because
the kernel doesn't feel like doing it :( . I'd kind of hoped to go in
the other direction if anything, with some kind of pseudo-write op
that let us swap a dirty shared_buffers entry from our shared_buffers
into the OS dirty buffer cache (on Linux at least) and let it handle
writeback, so we reduce double-buffering. Ha! So much for that!

-- 
 Craig Ringer                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services


Re: Postgres, fsync, and OSs (specifically linux)

From
Craig Ringer
Date:
On 28 April 2018 at 23:25, Simon Riggs <simon@2ndquadrant.com> wrote:
> On 27 April 2018 at 15:28, Andres Freund <andres@anarazel.de> wrote:
>
>> - Add a pre-checkpoint hook that checks for filesystem errors *after*
>>   fsyncing all the files, but *before* logging the checkpoint completion
>>   record. Operating systems, filesystems, etc. all log the error format
>>   differently, but for larger installations it'd not be too hard to
>>   write code that checks their specific configuration.
>>
>>   While I'm a bit concerned adding user-code before a checkpoint, if
>>   we'd do it as a shell command it seems pretty reasonable. And useful
>>   even without concern for the fsync issue itself. Checking for IO
>>   errors could e.g. also include checking for read errors - it'd not be
>>   unreasonable to not want to complete a checkpoint if there'd been any
>>   media errors.
>
> It seems clear that we need to evaluate our compatibility not just
> with an OS, as we do now, but with an OS/filesystem.
>
> Although people have suggested some approaches, I'm more interested in
> discovering how we can be certain we got it right.

TBH, we can't be certain, because there are too many failure modes,
some of which we can't really simulate in practical ways, or automated
ways.

But there are definitely steps we can take:

- Test the stack of FS, LVM (if any) etc with the dmsetup 'flakey'
target and a variety of workloads designed to hit errors at various
points. Some form of torture test.

- Almost up the device and see what happens if we write() then fsync()
enough to fill it.

- Plug-pull storage and see what happens, especially for multipath/iSCSI/SAN.

Experience with pg_test_fsync shows that it can also be hard to
reliably interpret the results of tests.

Again I'd like to emphasise that this is really only a significant
risk for a few configurations. Yes, it could result in Pg not failing
a checkpoint when it should if, say, your disk has a bad block it
can't repair and remap. But as Andres has pointed out in the past,
those sorts local storage failure cases tend toward "you're kind of
screwed anyway". It's only a serious concern when I/O errors are part
of the storage's accepted operation, as in multipath with default
settings.

We _definitely_ need to warn multipath users that the defaults are insane.

>> - Use direct IO. Due to architectural performance issues in PG and the
>>   fact that it'd not be applicable for all installations I don't think
>>   this is a reasonable fix for the issue presented here. Although it's
>>   independently something we should work on.  It might be worthwhile to
>>   provide a configuration that allows to force DIO to be enabled for WAL
>>   even if replication is turned on.
>
> "Use DirectIO" is roughly same suggestion as "don't trust Linux filesystems".

Surprisingly, that seems to be a lot of what's coming out of Linux
developers. Reliable buffered I/O? Why would you try to do that?

I know that's far from a universal position, though, and it sounds
like things were more productive in Andres's discussions at the meet.

-- 
 Craig Ringer                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services


Re: Postgres, fsync, and OSs (specifically linux)

From
Simon Riggs
Date:
On 28 April 2018 at 08:25, Simon Riggs <simon@2ndquadrant.com> wrote:
> On 27 April 2018 at 15:28, Andres Freund <andres@anarazel.de> wrote:
>
>> - Add a pre-checkpoint hook that checks for filesystem errors *after*
>>   fsyncing all the files, but *before* logging the checkpoint completion
>>   record. Operating systems, filesystems, etc. all log the error format
>>   differently, but for larger installations it'd not be too hard to
>>   write code that checks their specific configuration.
>>
>>   While I'm a bit concerned adding user-code before a checkpoint, if
>>   we'd do it as a shell command it seems pretty reasonable. And useful
>>   even without concern for the fsync issue itself. Checking for IO
>>   errors could e.g. also include checking for read errors - it'd not be
>>   unreasonable to not want to complete a checkpoint if there'd been any
>>   media errors.
>
> It seems clear that we need to evaluate our compatibility not just
> with an OS, as we do now, but with an OS/filesystem.
>
> Although people have suggested some approaches, I'm more interested in
> discovering how we can be certain we got it right.
>
> And the end result seems to be that PostgreSQL will be forced, in the
> short term, to declare certain combinations of OS/filesystem
> unsupported, with clear warning sent out to users.
>
> Adding a pre-checkpoint hook encourages people to fix this themselves
> without reporting issues, so I initially oppose this until we have a
> clearer argument as to why we need it. The answer is not to make this
> issue more obscure, but to make it more public.

Thinking some more, I think I understand, but please explain if not.

We need behavior that varies according to OS and filesystem, which
varies per tablespace.

We could have that variable behavior using

* a hook

* a set of GUC parameters that can be set at tablespace level

* a separate config file for each tablespace

My preference would be to avoid a hook.

-- 
Simon Riggs                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services


Re: Postgres, fsync, and OSs (specifically linux)

From
Simon Riggs
Date:
On 28 April 2018 at 09:15, Andres Freund <andres@anarazel.de> wrote:
> Hi,
>
> On 2018-04-28 08:25:53 -0700, Simon Riggs wrote:
>> > - Use direct IO. Due to architectural performance issues in PG and the
>> >   fact that it'd not be applicable for all installations I don't think
>> >   this is a reasonable fix for the issue presented here. Although it's
>> >   independently something we should work on.  It might be worthwhile to
>> >   provide a configuration that allows to force DIO to be enabled for WAL
>> >   even if replication is turned on.
>>
>> "Use DirectIO" is roughly same suggestion as "don't trust Linux filesystems".
>
> I want to emphasize that this is NOT a linux only issue. It's a problem
> across a number of operating systems, including linux.

Yes, of course.

>> It would be a major admission of defeat for us to take that as our
>> main route to a solution.
>
> Well, I think we were wrong to not engineer towards DIO. There's just
> too many issues with buffered IO to not have a supported path for
> DIO. But given that it's unrealistic to do so without major work, and
> wouldn't be applicable for all installations (shared_buffer size becomes
> critical), I don't think it matters that much for the issue discussed
> here.
>
>
>> The people I've spoken to so far have encouraged us to continue
>> working with the filesystem layer, offering encouragement of our
>> decision to use filesystems.
>
> There's a lot of people disagreeing with it too.

Specific recent verbal feedback from OpenLDAP was that the project
adopted DIO and found no benefit in doing so, with regret the other
way from having tried.

The care we need to use for any technique is the same.

-- 
Simon Riggs                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services


Re: Postgres, fsync, and OSs (specifically linux)

From
Thomas Munro
Date:
On Sun, Apr 29, 2018 at 10:42 PM, Simon Riggs <simon@2ndquadrant.com> wrote:
> On 28 April 2018 at 09:15, Andres Freund <andres@anarazel.de> wrote:
>> On 2018-04-28 08:25:53 -0700, Simon Riggs wrote:
>>> The people I've spoken to so far have encouraged us to continue
>>> working with the filesystem layer, offering encouragement of our
>>> decision to use filesystems.
>>
>> There's a lot of people disagreeing with it too.
>
> Specific recent verbal feedback from OpenLDAP was that the project
> adopted DIO and found no benefit in doing so, with regret the other
> way from having tried.

I'm not sure if OpenLDAP is really comparable.  The big three RDBMSs +
MySQL started like us and eventually switched to direct IO, I guess at
a time when direct IO support matured in OSs and their own IO
scheduling was thought to be superior.  I'm pretty sure they did that
because they didn't like wasting RAM on double buffering and had
better ideas about IO scheduling.  From some googling this morning:

DB2: The Linux/Unix/Windows edition changed its default to DIO ("NO
FILESYSTEM CACHING") in release 9.5 in 2007[1], but it can still do
buffered IO if you ask for it.

Oracle: Around the same time or earlier, in the Linux 2.4 era, Oracle
apparently supported direct IO ("FILESYSTEMIO_OPTIONS = DIRECTIO" (or
SETALL for DIRECTIO + ASYNCH)) on big iron Unix but didn't yet use it
on Linux[2].  There were some amusing emails from Linus Torvalds on
this topic[3].  I'm not sure what FILESYSTEMIO_OPTIONS's default value
is on each operating system today or when it changed, it's probably
SETALL everywhere by now?  I wonder if they stuck with buffered IO for
a time on Linux despite the availability of direct IO because they
thought it was more reliable or more performant.

SQL Server: I couldn't find any evidence that they've even kept the
option to use buffered IO (which must have existed in the ancestral
code base).  Can it?  It's a different situation though, targeting a
reduced set of platforms.

MySQL: The default is still buffered ("innodb_flush_method = fsync" as
opposed to "O_DIRECT") but O_DIRECT is supported and widely
recommended, so it sounds like it's usually a win.  Maybe not on
smaller systems though?

On MySQL, there are anecdotal reports of performance suffering on some
systems when you turn on O_DIRECT however.  If that's true, it's
interesting to speculate about why that might be as it would probably
apply also to us in early versions (optimistic explanation: the
kernel's stretchy page cache allows people to get away with poorly
tuned buffer pool size?  pessimistic explanation: the page reclamation
or IO scheduling (asynchronous write-back, write clustering,
read-ahead etc) is not as good as the OS's, but that effect is hidden
by suitably powerful disk subsystem with its own magic caching?)  Note
that its O_DIRECT setting *also* calls fsync() to flush filesystem
meta-data (necessary if the file was extended); I wonder if that is
exposed to write-back error loss.

[1] https://www.ibm.com/support/knowledgecenter/en/SSEPGG_9.5.0/com.ibm.db2.luw.admin.dbobj.doc/doc/c0051304.html
[2] http://www.ixora.com.au/notes/direct_io.htm
[3] https://lkml.org/lkml/2002/5/11/58

-- 
Thomas Munro
http://www.enterprisedb.com


Re: Postgres, fsync, and OSs (specifically linux)

From
Thomas Munro
Date:
On Mon, Apr 30, 2018 at 11:02 AM, Thomas Munro
<thomas.munro@enterprisedb.com> wrote:
> MySQL: The default is still buffered

Someone pulled me up on this off-list: the default is buffered (fsync)
on Unix, but it's unbuffered on Windows.  That's quite interesting.

https://dev.mysql.com/doc/refman/8.0/en/innodb-parameters.html#sysvar_innodb_flush_method
https://mariadb.com/kb/en/library/xtradbinnodb-server-system-variables/#innodb_flush_method

-- 
Thomas Munro
http://www.enterprisedb.com


Re: Postgres, fsync, and OSs (specifically linux)

From
Thomas Munro
Date:
On Sun, Apr 29, 2018 at 1:58 PM, Craig Ringer <craig@2ndquadrant.com> wrote:
> On 28 April 2018 at 23:25, Simon Riggs <simon@2ndquadrant.com> wrote:
>> On 27 April 2018 at 15:28, Andres Freund <andres@anarazel.de> wrote:
>>>   While I'm a bit concerned adding user-code before a checkpoint, if
>>>   we'd do it as a shell command it seems pretty reasonable. And useful
>>>   even without concern for the fsync issue itself. Checking for IO
>>>   errors could e.g. also include checking for read errors - it'd not be
>>>   unreasonable to not want to complete a checkpoint if there'd been any
>>>   media errors.
>>
>> It seems clear that we need to evaluate our compatibility not just
>> with an OS, as we do now, but with an OS/filesystem.
>>
>> Although people have suggested some approaches, I'm more interested in
>> discovering how we can be certain we got it right.
>
> TBH, we can't be certain, because there are too many failure modes,
> some of which we can't really simulate in practical ways, or automated
> ways.

+1

Testing is good, but unless you have a categorical statement from the
relevant documentation or kernel team or you have the source code, I'm
not sure how you can ever really be sure about this.  I think we have
a fair idea now what several open kernels do, but we still haven't got
a clue about Windows, AIX, HPUX and Solaris and we only have half the
answer for Illumos, and no "negative" test result can prove that they
can't throw away write-back errors or data.

Considering the variety in interpretation and liberties taken, I
wonder if fsync() is underspecified and someone should file an issue
over at http://www.opengroup.org/austin/ about that.

-- 
Thomas Munro
http://www.enterprisedb.com


Re: Postgres, fsync, and OSs (specifically linux)

From
Craig Ringer
Date:
On 30 April 2018 at 09:09, Thomas Munro <thomas.munro@enterprisedb.com> wrote:

> Considering the variety in interpretation and liberties taken, I
> wonder if fsync() is underspecified and someone should file an issue
> over at http://www.opengroup.org/austin/ about that.

All it's going to achieve is adding an "is implementation-defined"
caveat, but that's at least a bit of a heads-up.

I filed patches for Linux man-pages ages ago. I'll update them and
post to LKML; apparently bugzilla has a lot of spam and many people
ignore notifications, so they might just bitrot forever otherwise.

Meanwhile, do we know if, on Linux 4.13+, if we get a buffered write
error due to dirty writeback before we close() a file we don't
fsync(), we'll get the error on close()?

-- 
 Craig Ringer                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services


Re: Postgres, fsync, and OSs (specifically linux)

From
Andres Freund
Date:
On 2018-04-30 10:14:23 +0800, Craig Ringer wrote:
> Meanwhile, do we know if, on Linux 4.13+, if we get a buffered write
> error due to dirty writeback before we close() a file we don't
> fsync(), we'll get the error on close()?

Not quite sure what you're getting at with "a file we don't fsync" - if
we don't, we don't care about durability anyway, no? Or do you mean
where we fsync in a different process?

Either way, the answer is mostly no: On NFS et al where close() implies
an fsync you'll get the error at that time, otherwise you'll get it at
the next fsync().

Greetings,

Andres Freund


Re: Postgres, fsync, and OSs (specifically linux)

From
Craig Ringer
Date:
> Not quite sure what you're getting at with "a file we don't fsync" - if
> we don't, we don't care about durability anyway, no? Or do you mean
> where we fsync in a different process?

Right.

> Either way, the answer is mostly no: On NFS et al where close() implies
> an fsync you'll get the error at that time, otherwise you'll get it at
> the next fsync().

Thanks.

The reason I ask is that if we got notified of already-detected
writeback errors (on 4.13+) on close() too, it'd narrow the window a
little for problems, since normal backends could PANIC if close() of a
persistent file raised EIO. Otherwise we're less likely to see the
error, since the checkpointer won't see it - it happened before the
checkpointer open()ed the file. It'd still be no help for dirty
writeback that happens after we close() in a user backend / the
bgwriter and before we re-open(), but it'd be nice if the kernel would
tell us on close() if it knows of a writeback error.

-- 
 Craig Ringer                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services


Re: Postgres, fsync, and OSs (specifically linux)

From
Craig Ringer
Date:
Hrm, something else that just came up. On 9.6+ we use sync_file_range.
It's surely going to eat errors:

        rc = sync_file_range(fd, offset, nbytes,
                             SYNC_FILE_RANGE_WRITE);

        /* don't error out, this is just a performance optimization */
        if (rc != 0)
        {
            ereport(WARNING,
                    (errcode_for_file_access(),
                     errmsg("could not flush dirty data: %m")));
        }

so that has to panic too.

I'm very suspicious about the safety of the msync() path too.

I'll post an update to my PANIC-everywhere patch that add these cases.


Re: Postgres, fsync, and OSs (specifically linux)

From
Andres Freund
Date:
On 2018-04-30 13:03:24 +0800, Craig Ringer wrote:
> Hrm, something else that just came up. On 9.6+ we use sync_file_range.
> It's surely going to eat errors:
> 
>         rc = sync_file_range(fd, offset, nbytes,
>                              SYNC_FILE_RANGE_WRITE);
> 
>         /* don't error out, this is just a performance optimization */
>         if (rc != 0)
>         {
>             ereport(WARNING,
>                     (errcode_for_file_access(),
>                      errmsg("could not flush dirty data: %m")));
>         }

It's not. Only SYNC_FILE_RANGE_WAIT_{BEFORE,AFTER} eat errors. Which
seems sensible, because they could be considered data integrity
operations.

fs/sync.c:
SYSCALL_DEFINE4(sync_file_range, int, fd, loff_t, offset, loff_t, nbytes,
                unsigned int, flags)
{
...

    if (flags & SYNC_FILE_RANGE_WAIT_BEFORE) {
        ret = file_fdatawait_range(f.file, offset, endbyte);
        if (ret < 0)
            goto out_put;
    }

    if (flags & SYNC_FILE_RANGE_WRITE) {
        ret = __filemap_fdatawrite_range(mapping, offset, endbyte,
                         WB_SYNC_NONE);
        if (ret < 0)
            goto out_put;
    }

    if (flags & SYNC_FILE_RANGE_WAIT_AFTER)
        ret = file_fdatawait_range(f.file, offset, endbyte);


where
int file_fdatawait_range(struct file *file, loff_t start_byte, loff_t end_byte)
{
    struct address_space *mapping = file->f_mapping;

    __filemap_fdatawait_range(mapping, start_byte, end_byte);
    return file_check_and_advance_wb_err(file);
}
EXPORT_SYMBOL(file_fdatawait_range);

the critical call is file_check_and_advance_wb_err(). That's the call
that checks and clears errors.

Would be good to add a kernel xfstest (gets used on all relevant FS
these days), to make sure that doesn't change.


> I'm very suspicious about the safety of the msync() path too.

That seems justified however:

SYSCALL_DEFINE3(msync, unsigned long, start, size_t, len, int, flags)
{
...
            error = vfs_fsync_range(file, fstart, fend, 1);
..


 */
int vfs_fsync_range(struct file *file, loff_t start, loff_t end, int datasync)
{
...
    return file->f_op->fsync(file, start, end, datasync);
}
EXPORT_SYMBOL(vfs_fsync_range);



int ext4_sync_file(struct file *file, loff_t start, loff_t end, int datasync)
{
...
    ret = file_write_and_wait_range(file, start, end);
    if (ret)
        return ret;
...


STATIC int
xfs_file_fsync(
    struct file        *file,
    loff_t            start,
    loff_t            end,
    int            datasync)
{
...
    error = file_write_and_wait_range(file, start, end);
    if (error)
        return error;


int file_write_and_wait_range(struct file *file, loff_t lstart, loff_t lend)
{
...
    err2 = file_check_and_advance_wb_err(file);
    if (!err)
        err = err2;
    return err;
}


Greetings,

Andres Freund


Re: Postgres, fsync, and OSs (specifically linux)

From
Craig Ringer
Date:
On 1 May 2018 at 00:09, Andres Freund <andres@anarazel.de> wrote:

> It's not. Only SYNC_FILE_RANGE_WAIT_{BEFORE,AFTER} eat errors. Which
> seems sensible, because they could be considered data integrity
> operations.

Ah, I misread that. Thankyou.

>> I'm very suspicious about the safety of the msync() path too.
>
> That seems justified however:

I'll add EIO tests there.

-- 
 Craig Ringer                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services


Re: Postgres, fsync, and OSs (specifically linux)

From
Catalin Iacob
Date:
On Sat, Apr 28, 2018 at 12:28 AM, Andres Freund <andres@anarazel.de> wrote:
> Before linux v4.13 errors in kernel writeback would be reported at most
> once, without a guarantee that that'd happen (IIUC memory pressure could
> lead to the relevant information being evicted) - but it was pretty
> likely.  After v4.13 (see https://lwn.net/Articles/724307/) errors are
> reported exactly once to all open file descriptors for a file with an
> error - but never for files that have been opened after the error
> occurred.

snip

> == Proposed Linux Changes ==
>
> - Matthew Wilcox proposed (and posted a patch) that'd partially revert
>   behaviour to the pre v4.13 world, by *also* reporting errors to
>   "newer" file-descriptors if the error hasn't previously been
>   reported. That'd still not guarantee that the error is reported
>   (memory pressure could evict information without open fd), but in most
>   situations we'll again get the error in the checkpointer.
>
>   This seems largely be agreed upon. It's unclear whether it'll go into
>   the stable backports for still-maintained >= v4.13 kernels.

This is now merged, if it's not reverted it will appear in v4.17.
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=fff75eb2a08c2ac96404a2d79685668f3cf5a7a3

The commit is cc-ed to stable so it should get picked up in the near future.
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=b4678df184b314a2bd47d2329feca2c2534aa12b


Re: Postgres, fsync, and OSs (specifically linux)

From
Andres Freund
Date:
Hi,

On 2018-04-27 15:28:42 -0700, Andres Freund wrote:
> I went to LSF/MM 2018 to discuss [0] and related issues. Overall I'd say
> it was a very productive discussion.  I'll first try to recap the
> current situation, updated with knowledge I gained. Secondly I'll try to
> discuss the kernel changes that seem to have been agreed upon. Thirdly
> I'll try to sum up what postgres needs to change.

LWN summarized the discussion as well:

https://lwn.net/SubscriberLink/752952/6825e6a1ddcfb1f3/

- Andres


Re: Postgres, fsync, and OSs (specifically linux)

From
Andres Freund
Date:
On 2018-05-01 09:38:03 +0800, Craig Ringer wrote:
> On 1 May 2018 at 00:09, Andres Freund <andres@anarazel.de> wrote:
> 
> > It's not. Only SYNC_FILE_RANGE_WAIT_{BEFORE,AFTER} eat errors. Which
> > seems sensible, because they could be considered data integrity
> > operations.
> 
> Ah, I misread that. Thankyou.
> 
> >> I'm very suspicious about the safety of the msync() path too.
> >
> > That seems justified however:
> 
> I'll add EIO tests there.

Do you have a patchset including that?  I didn't find anything after a
quick search...

Greetings,

Andres Freund


Re: Postgres, fsync, and OSs (specifically linux)

From
Craig Ringer
Date:
On 10 May 2018 at 06:55, Andres Freund <andres@anarazel.de> wrote:

> Do you have a patchset including that?  I didn't find anything after a
> quick search...

There was an earlier rev on the other thread but without msync checks.

I've added panic for msync in the attached, and tidied the comments a bit.

I didn't add comments on why we panic to each individual pg_fsync or
FileSync caller that panics; instead pg_fsync points to
pg_fsync_no_writethrough which explains it briefly and has links.

I looked at callers of pg_fsync, pg_fsync_no_writethrough,
pg_fsync_writethrough, mdsync, and FileSync when writing this.

WAL writing already PANIC'd on fsync failure, which helps, though we
now know that's not sufficient for complete safety.


Patch on top of v11 HEAD @ ddc1f32ee507

-- 
 Craig Ringer                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

Attachment

Re: Postgres, fsync, and OSs (specifically linux)

From
Andres Freund
Date:
Hi,

On 2018-05-10 09:50:03 +0800, Craig Ringer wrote:
>      while ((src = (RewriteMappingFile *) hash_seq_search(&seq_status)) != NULL)
>      {
>          if (FileSync(src->vfd, WAIT_EVENT_LOGICAL_REWRITE_SYNC) != 0)
> -            ereport(ERROR,
> +            ereport(PANIC,
>                      (errcode_for_file_access(),
>                       errmsg("could not fsync file \"%s\": %m", src->path)));

To me this (and the other callers) doesn't quite look right. First, I
think we should probably be a bit more restrictive about when PANIC
out. It seems like we should PANIC on ENOSPC and EIO, but possibly not
others.  Secondly, I think we should centralize the error handling. It
seems likely that we'll acrue some platform specific workarounds, and I
don't want to copy that knowledge everywhere.

Also, don't we need the same on close()?

- Andres


Re: Postgres, fsync, and OSs (specifically linux)

From
Robert Haas
Date:
On Thu, May 17, 2018 at 12:44 PM, Andres Freund <andres@anarazel.de> wrote:
> Hi,
>
> On 2018-05-10 09:50:03 +0800, Craig Ringer wrote:
>>       while ((src = (RewriteMappingFile *) hash_seq_search(&seq_status)) != NULL)
>>       {
>>               if (FileSync(src->vfd, WAIT_EVENT_LOGICAL_REWRITE_SYNC) != 0)
>> -                     ereport(ERROR,
>> +                     ereport(PANIC,
>>                                       (errcode_for_file_access(),
>>                                        errmsg("could not fsync file \"%s\": %m", src->path)));
>
> To me this (and the other callers) doesn't quite look right. First, I
> think we should probably be a bit more restrictive about when PANIC
> out. It seems like we should PANIC on ENOSPC and EIO, but possibly not
> others.  Secondly, I think we should centralize the error handling. It
> seems likely that we'll acrue some platform specific workarounds, and I
> don't want to copy that knowledge everywhere.

Maybe something like:

ereport(promote_eio_to_panic(ERROR), ...)

?

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company


Re: Postgres, fsync, and OSs (specifically linux)

From
Ashutosh Bapat
Date:
On Thu, May 17, 2018 at 11:45 PM, Robert Haas <robertmhaas@gmail.com> wrote:
> On Thu, May 17, 2018 at 12:44 PM, Andres Freund <andres@anarazel.de> wrote:
>> Hi,
>>
>> On 2018-05-10 09:50:03 +0800, Craig Ringer wrote:
>>>       while ((src = (RewriteMappingFile *) hash_seq_search(&seq_status)) != NULL)
>>>       {
>>>               if (FileSync(src->vfd, WAIT_EVENT_LOGICAL_REWRITE_SYNC) != 0)
>>> -                     ereport(ERROR,
>>> +                     ereport(PANIC,
>>>                                       (errcode_for_file_access(),
>>>                                        errmsg("could not fsync file \"%s\": %m", src->path)));
>>
>> To me this (and the other callers) doesn't quite look right. First, I
>> think we should probably be a bit more restrictive about when PANIC
>> out. It seems like we should PANIC on ENOSPC and EIO, but possibly not
>> others.  Secondly, I think we should centralize the error handling. It
>> seems likely that we'll acrue some platform specific workarounds, and I
>> don't want to copy that knowledge everywhere.
>
> Maybe something like:
>
> ereport(promote_eio_to_panic(ERROR), ...)

Well, searching for places where error is reported with level PANIC,
using word PANIC would miss these instances. People will have to
remember to search with promote_eio_to_panic. May be handle the errors
inside FileSync() itself or a wrapper around that.

-- 
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company


Re: Postgres, fsync, and OSs (specifically linux)

From
Andres Freund
Date:
Hi,

On 2018-04-27 15:28:42 -0700, Andres Freund wrote:
> == Potential Postgres Changes ==
> 
> Several operating systems / file systems behave differently (See
> e.g. [2], thanks Thomas) than we expected. Even the discussed changes to
> e.g. linux don't get to where we thought we are. There's obviously also
> the question of how to deal with kernels / OSs that have not been
> updated.
> 
> Changes that appear to be necessary, even for kernels with the issues
> addressed:
> 
> - Clearly we need to treat fsync() EIO, ENOSPC errors as a PANIC and
>   retry recovery. While ENODEV (underlying device went away) will be
>   persistent, it probably makes sense to treat it the same or even just
>   give up and shut down.  One question I see here is whether we just
>   want to continue crash-recovery cycles, or whether we want to limit
>   that.

Craig has a patch for this, although I'm not yet 100% happy with it.


> - We need more aggressive error checking on close(), for ENOSPC and
>   EIO. In both cases afaics we'll have to trigger a crash recovery
>   cycle. It's entirely possible to end up in a loop on NFS etc, but I
>   don't think there's a way around that.

This needs to be handled.


> - The outstanding fsync request queue isn't persisted properly [3]. This
>   means that even if the kernel behaved the way we'd expected, we'd not
>   fail a second checkpoint :(. It's possible that we don't need to deal
>   with this because we'll henceforth PANIC, but I'd argue we should fix
>   that regardless. Seems like a time-bomb otherwise (e.g. after moving
>   to DIO somebody might want to relax the PANIC...).

> What we could do:
> 
> - forward file descriptors from backends to checkpointer (using
>   SCM_RIGHTS) when marking a segment dirty. That'd require some
>   optimizations (see [4]) to avoid doing so repeatedly.  That'd
>   guarantee correct behaviour in all linux kernels >= 4.13 (possibly
>   backported by distributions?), and I think it'd also make it vastly
>   more likely that errors are reported in earlier kernels.
> 
>   This should be doable without a noticeable performance impact, I
>   believe.  I don't think it'd be that hard either, but it'd be a bit of
>   a pain to backport it to all postgres versions, as well as a bit
>   invasive for that.
> 
>   The infrastructure this'd likely end up building (hashtable of open
>   relfilenodes), would likely be useful for further things (like caching
>   file size).

I've written a patch series for this. Took me quite a bit longer than I
had hoped.

The attached patchseries consists out of a few preparatory patches:
- freespace optimization to not call smgrexists() unnecessarily
- register_dirty_segment() optimization to not queue requests for
  segments that locally are known to already have been dirtied.  This
  seems like a good optimization regardless of further changes.  Doesn't
  yet deal with the mdsync counter wrapping around (which is unlikely to
  ever happen in practice, it's 32bit).
- some fd.c changes, I don't think they're quite right yet
- new functions to send/recv data over a unix domain socket, *including*
  a file descriptor.

The main patch guarantees that fsync requests are forwarded from
backends to the checkpointer, including the file descriptor. As we do so
immediately at mdwrite() time, that guarantees that the fd has been open
from before the write started, therefore linux will guarantee that that
FD will see errors.

The design of the patch went through a few iterations. I initially
attempted to make the fsync request hashtable shared, but that turned
out to be a lot harder to do reliably *and* fast than I was anticipating
(we'd need to hold a lock for the entirety of mdsync(), dynahash doesn't
allow iteration while other backends modify).

So what I instead did was to replace the shared memory fsync request
queue with a unix domain socket (created in postmaster, using
socketpair()).  CheckpointerRequest structs are written to that queue,
including the associated file descriptor.  The checkpointer absorbs
those requests, and updates the local pending requests hashtable in
local process memory.  To facilitate that mdsync() has read all requests
from the last cycle, checkpointer self-enqueues a token, which allows
to detect the end of the relevant portion of the queue.

The biggest complication in all this scheme is that checkpointer now
needs to keep a file descriptor open for every segment. That obviously
requires adding a few new fields to the hashtable entry. But the bigger
issue is that it's now possible that pending requests need to be
processed earlier than the next checkpoint, because of file descriptor
limits.  To address that absorbing the fsync request queue will now do a
mdsync() style pass, doing the necessary fsyncs.

Because mdsync() (or rather its new workhorse mdsyncpass()) will now not
open files itself, there's no need to do deal with retries for files
that have been deleted. For the cases where we didn't yet receive a
fsync cancel request, we'll just fsync the fd. That's unnecessary, but
harmless.


Obviously this is currently heavily unix specific (according to my
research all our unix platforms support say that they support sending
fds across unix domain sockets w/ SCM_RIGHTS).  It's unclear whether any
OS but linux benefits from not closing file descriptors before fsync().

We could make this work for windows, without *too* much trouble (one can
just open fds in another process, using that process' handle).

I think there's some advantage in using the same approach
everywhere. For one not maintaining two radically different approaches
for complicated code.  It'd also allow us to offload more fsyncs to
checkpointer, not just the ones for normal relation files, which does
seem advantageous. Not having ugly retry logic around deleted files in
mdsync() also seems nice.  But there's cases where this is likely
slower, due to the potential of having to wait for checkpointer when the
queue is full.

I'll note that I think the new mdsync() is considerably simpler. Even if
we do not decide to use an approach as presented here, I think we should
make some of those changes. Specifically not unlinking the pending
requests bitmap in mdsync() seems like it both resolves existing bug
(see upthread) and makes the code simpler.


I plan to switch to working on something else for a day or two next
week, and then polish this further. I'd greatly appreciate comments till
then.


I didn't want to do this now, but I think we should also consider
removing all awareness of segments from the fsync request queue. Instead
it should deal with individual files, and the segmentation should be
handled by md.c.  That'll allow us to move all the necessary code to
smgr.c (or checkpointer?); Thomas said that'd be helpful for further
work.  I personally think it'd be a lot simpler, because having to have
long bitmaps with only the last bit set for large append only relations
isn't a particularly sensible approach imo.  The only thing that that'd
make more complicated is that the file/database unlink requests get more
expensive (as they'd likely need to search the whole table), but that
seems like a sensible tradeoff. Alternatively using a tree structure
would be an alternative obviously.  Personally I was thinking that we
should just make the hashtable be over a pathname, that seems most
generic.

Greetings,

Andres Freund

Attachment

Re: Postgres, fsync, and OSs (specifically linux)

From
Stephen Frost
Date:
Greetings,

* Ashutosh Bapat (ashutosh.bapat@enterprisedb.com) wrote:
> On Thu, May 17, 2018 at 11:45 PM, Robert Haas <robertmhaas@gmail.com> wrote:
> > On Thu, May 17, 2018 at 12:44 PM, Andres Freund <andres@anarazel.de> wrote:
> >> Hi,
> >>
> >> On 2018-05-10 09:50:03 +0800, Craig Ringer wrote:
> >>>       while ((src = (RewriteMappingFile *) hash_seq_search(&seq_status)) != NULL)
> >>>       {
> >>>               if (FileSync(src->vfd, WAIT_EVENT_LOGICAL_REWRITE_SYNC) != 0)
> >>> -                     ereport(ERROR,
> >>> +                     ereport(PANIC,
> >>>                                       (errcode_for_file_access(),
> >>>                                        errmsg("could not fsync file \"%s\": %m", src->path)));
> >>
> >> To me this (and the other callers) doesn't quite look right. First, I
> >> think we should probably be a bit more restrictive about when PANIC
> >> out. It seems like we should PANIC on ENOSPC and EIO, but possibly not
> >> others.  Secondly, I think we should centralize the error handling. It
> >> seems likely that we'll acrue some platform specific workarounds, and I
> >> don't want to copy that knowledge everywhere.
> >
> > Maybe something like:
> >
> > ereport(promote_eio_to_panic(ERROR), ...)
>
> Well, searching for places where error is reported with level PANIC,
> using word PANIC would miss these instances. People will have to
> remember to search with promote_eio_to_panic. May be handle the errors
> inside FileSync() itself or a wrapper around that.

No, that search wouldn't miss those instances- such a search would find
promote_eio_to_panic() and then someone would go look up the uses of
that function.  That hardly seems like a serious issue for folks hacking
on PG.

I'm not saying that having a wrapper around FileSync() would be bad or
having it handle things, but I don't agree with the general notion that
we can't have a function which handles the complicated bits about the
kind of error because someone grep'ing the source for PANIC might have
to do an additional lookup.

Thanks!

Stephen

Attachment

Re: Postgres, fsync, and OSs (specifically linux)

From
Abhijit Menon-Sen
Date:
At 2018-05-18 20:27:57 -0400, sfrost@snowman.net wrote:
>
> I don't agree with the general notion that we can't have a function
> which handles the complicated bits about the kind of error because
> someone grep'ing the source for PANIC might have to do an additional
> lookup.

Or we could just name the function promote_eio_to_PANIC.

(I understood the objection to be about how 'grep PANIC' wouldn't find
these lines at all, not that there would be an additional lookup.)

-- Abhijit


Re: Postgres, fsync, and OSs (specifically linux)

From
Stephen Frost
Date:
Greetings,

* Abhijit Menon-Sen (ams@2ndQuadrant.com) wrote:
> At 2018-05-18 20:27:57 -0400, sfrost@snowman.net wrote:
> >
> > I don't agree with the general notion that we can't have a function
> > which handles the complicated bits about the kind of error because
> > someone grep'ing the source for PANIC might have to do an additional
> > lookup.
>
> Or we could just name the function promote_eio_to_PANIC.

Ugh, I'm not thrilled with that either.

> (I understood the objection to be about how 'grep PANIC' wouldn't find
> these lines at all, not that there would be an additional lookup.)

... and my point was that 'grep PANIC' would, almost certainly, find the
function promote_eio_to_panic(), and someone could trivially look up all
the callers of that function then.

Thanks!

Stephen

Attachment

Re: Postgres, fsync, and OSs (specifically linux)

From
Thomas Munro
Date:
On Sat, May 19, 2018 at 9:03 AM, Andres Freund <andres@anarazel.de> wrote:
> I've written a patch series for this. Took me quite a bit longer than I
> had hoped.

Great.

> I plan to switch to working on something else for a day or two next
> week, and then polish this further. I'd greatly appreciate comments till
> then.

Took it for a spin on macOS and FreeBSD.  First problem:

+       if (socketpair(AF_UNIX, SOCK_STREAM | SOCK_CLOEXEC, 0, fsync_fds) < 0)

SOCK_CLOEXEC isn't portable (FreeBSD yes since 10, macOS no, others
who knows).  Adding FD_CLOEXEC to your later fcntl() calls is probably
the way to do it?  I understand from reading the Linux man pages that
there are race conditions with threads but that doesn't apply here.

Next, make check hangs in initdb on both of my pet OSes when md.c
raises an error (fseek fails) and we raise and error while raising and
error and deadlock against ourselves.  Backtrace here:
https://paste.debian.net/1025336/

Apparently the initial error was that mdextend() called _mdnblocks()
which called FileSeek() on vfd 43 == fd 30, pathname "base/1/2704",
but when I check my operating system open file descriptor table I find
that there is no fd 30: there is a 29 and a 31, so it has already been
unexpectedly closed.

I could dig further and/or provide a shell on a system with dev tools.

> I didn't want to do this now, but I think we should also consider
> removing all awareness of segments from the fsync request queue. Instead
> it should deal with individual files, and the segmentation should be
> handled by md.c.  That'll allow us to move all the necessary code to
> smgr.c (or checkpointer?); Thomas said that'd be helpful for further
> work.  I personally think it'd be a lot simpler, because having to have
> long bitmaps with only the last bit set for large append only relations
> isn't a particularly sensible approach imo.  The only thing that that'd
> make more complicated is that the file/database unlink requests get more
> expensive (as they'd likely need to search the whole table), but that
> seems like a sensible tradeoff. Alternatively using a tree structure
> would be an alternative obviously.  Personally I was thinking that we
> should just make the hashtable be over a pathname, that seems most
> generic.

+1

I'll be posting a patch shortly that also needs similar machinery, but
can't easily share with md.c due to technical details.  I'd love there
to be just one of those, and for it to be simpler and general.

-- 
Thomas Munro
http://www.enterprisedb.com


Re: Postgres, fsync, and OSs (specifically linux)

From
Thomas Munro
Date:
On Sat, May 19, 2018 at 4:51 PM, Thomas Munro
<thomas.munro@enterprisedb.com> wrote:
> Next, make check hangs in initdb on both of my pet OSes when md.c
> raises an error (fseek fails) and we raise and error while raising and
> error and deadlock against ourselves.  Backtrace here:
> https://paste.debian.net/1025336/

Ah, I see now that something similar is happening on Linux too, so I
guess you already knew this.

https://travis-ci.org/postgresql-cfbot/postgresql/builds/380913223

-- 
Thomas Munro
http://www.enterprisedb.com


Re: Postgres, fsync, and OSs (specifically linux)

From
Ashutosh Bapat
Date:
On Sat, May 19, 2018 at 6:31 AM, Stephen Frost <sfrost@snowman.net> wrote:
> Greetings,
>
> * Abhijit Menon-Sen (ams@2ndQuadrant.com) wrote:
>> At 2018-05-18 20:27:57 -0400, sfrost@snowman.net wrote:
>> >
>> > I don't agree with the general notion that we can't have a function
>> > which handles the complicated bits about the kind of error because
>> > someone grep'ing the source for PANIC might have to do an additional
>> > lookup.
>>
>> Or we could just name the function promote_eio_to_PANIC.
>
> Ugh, I'm not thrilled with that either.
>
>> (I understood the objection to be about how 'grep PANIC' wouldn't find
>> these lines at all, not that there would be an additional lookup.)
>
> ... and my point was that 'grep PANIC' would, almost certainly, find the
> function promote_eio_to_panic(), and someone could trivially look up all
> the callers of that function then.

It's not just grep,  but tools like cscope, tag. Although, I agree,
that adding a function, if all necessary, is more important than
convenience of finding all the instances of a certain token easily.

-- 
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company


Re: Postgres, fsync, and OSs (specifically linux)

From
Craig Ringer
Date:
On 18 May 2018 at 00:44, Andres Freund <andres@anarazel.de> wrote:
Hi,

On 2018-05-10 09:50:03 +0800, Craig Ringer wrote:
>       while ((src = (RewriteMappingFile *) hash_seq_search(&seq_status)) != NULL)
>       {
>               if (FileSync(src->vfd, WAIT_EVENT_LOGICAL_REWRITE_SYNC) != 0)
> -                     ereport(ERROR,
> +                     ereport(PANIC,
>                                       (errcode_for_file_access(),
>                                        errmsg("could not fsync file \"%s\": %m", src->path)));

To me this (and the other callers) doesn't quite look right. First, I
think we should probably be a bit more restrictive about when PANIC
out. It seems like we should PANIC on ENOSPC and EIO, but possibly not
others.  Secondly, I think we should centralize the error handling. It
seems likely that we'll acrue some platform specific workarounds, and I
don't want to copy that knowledge everywhere.

Also, don't we need the same on close()?


Yes, we do, and that expands the scope a bit.

I agree with Robert that some sort of filter/macro is wise, though naming it clearly will be tricky.

I'll have a look.

--
 Craig Ringer                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

Re: Postgres, fsync, and OSs (specifically linux)

From
Craig Ringer
Date:
On 21 May 2018 at 12:57, Craig Ringer <craig@2ndquadrant.com> wrote:
On 18 May 2018 at 00:44, Andres Freund <andres@anarazel.de> wrote:
Hi,

On 2018-05-10 09:50:03 +0800, Craig Ringer wrote:
>       while ((src = (RewriteMappingFile *) hash_seq_search(&seq_status)) != NULL)
>       {
>               if (FileSync(src->vfd, WAIT_EVENT_LOGICAL_REWRITE_SYNC) != 0)
> -                     ereport(ERROR,
> +                     ereport(PANIC,
>                                       (errcode_for_file_access(),
>                                        errmsg("could not fsync file \"%s\": %m", src->path)));

To me this (and the other callers) doesn't quite look right. First, I
think we should probably be a bit more restrictive about when PANIC
out. It seems like we should PANIC on ENOSPC and EIO, but possibly not
others.  Secondly, I think we should centralize the error handling. It
seems likely that we'll acrue some platform specific workarounds, and I
don't want to copy that knowledge everywhere.

Also, don't we need the same on close()?


Yes, we do, and that expands the scope a bit.

I agree with Robert that some sort of filter/macro is wise, though naming it clearly will be tricky.

I'll have a look.


On the queue for tomorrow. 


--
 Craig Ringer                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

Re: Postgres, fsync, and OSs (specifically linux)

From
Andres Freund
Date:
On 2018-05-19 18:12:52 +1200, Thomas Munro wrote:
> On Sat, May 19, 2018 at 4:51 PM, Thomas Munro
> <thomas.munro@enterprisedb.com> wrote:
> > Next, make check hangs in initdb on both of my pet OSes when md.c
> > raises an error (fseek fails) and we raise and error while raising and
> > error and deadlock against ourselves.  Backtrace here:
> > https://paste.debian.net/1025336/
> 
> Ah, I see now that something similar is happening on Linux too, so I
> guess you already knew this.

I didn't. I cleaned something up and only tested installcheck
after... Singleuser mode was broken.

Attached is a new version.

I've changed my previous attempt at using transient files to using File
type files, but unliked from the LRU so that they're kept open. Not sure
if that's perfect, but seems cleaner.

Greetings,

Andres Freund

Attachment

Re: Postgres, fsync, and OSs (specifically linux)

From
Dmitry Dolgov
Date:
> On 22 May 2018 at 03:08, Andres Freund <andres@anarazel.de> wrote:
> On 2018-05-19 18:12:52 +1200, Thomas Munro wrote:
>> On Sat, May 19, 2018 at 4:51 PM, Thomas Munro
>> <thomas.munro@enterprisedb.com> wrote:
>> > Next, make check hangs in initdb on both of my pet OSes when md.c
>> > raises an error (fseek fails) and we raise and error while raising and
>> > error and deadlock against ourselves.  Backtrace here:
>> > https://paste.debian.net/1025336/
>>
>> Ah, I see now that something similar is happening on Linux too, so I
>> guess you already knew this.
>
> I didn't. I cleaned something up and only tested installcheck
> after... Singleuser mode was broken.
>
> Attached is a new version.
>
> I've changed my previous attempt at using transient files to using File
> type files, but unliked from the LRU so that they're kept open. Not sure
> if that's perfect, but seems cleaner.

Thanks for the patch. Out of curiosity I tried to play with it a bit.
`pgbench -i -s 100` actually hang on my machine, because the
copy process ended up with waiting after `pg_uds_send_with_fd`
had

    errno == EWOULDBLOCK || errno == EAGAIN

as well as the checkpointer process. Looks like with the default
configuration and `max_wal_size=1GB` it writes more than reads to a
socket, and a buffer eventually becomes full. I've increased
SO_RCVBUF/SO_SNDBUF and `max_wal_size` independently to
check it, and in both cases the problem disappeared (but I assume
only for this particular scale). Is it something that was already considered?


Re: Postgres, fsync, and OSs (specifically linux)

From
Andres Freund
Date:
Hi,


On 2018-05-22 17:37:28 +0200, Dmitry Dolgov wrote:
> Thanks for the patch. Out of curiosity I tried to play with it a bit.

Thanks.


> `pgbench -i -s 100` actually hang on my machine, because the
> copy process ended up with waiting after `pg_uds_send_with_fd`
> had

Hm, that had worked at some point...


>     errno == EWOULDBLOCK || errno == EAGAIN
> 
> as well as the checkpointer process.

What do you mean with that latest sentence?


> Looks like with the default
> configuration and `max_wal_size=1GB` it writes more than reads to a
> socket, and a buffer eventually becomes full.

That's intended to then wake up the checkpointer immediately, so it can
absorb the requests.  So something isn't right yet.


> I've increased SO_RCVBUF/SO_SNDBUF and `max_wal_size` independently to
> check it, and in both cases the problem disappeared (but I assume only
> for this particular scale). Is it something that was already
> considered?

It's considered. Tuning up those might help with performance, but
shouldn't required from a correctness POV.  Hm.

Greetings,

Andres Freund


Re: Postgres, fsync, and OSs (specifically linux)

From
Andres Freund
Date:
On 2018-05-22 08:57:18 -0700, Andres Freund wrote:
> Hi,
> 
> 
> On 2018-05-22 17:37:28 +0200, Dmitry Dolgov wrote:
> > Thanks for the patch. Out of curiosity I tried to play with it a bit.
> 
> Thanks.
> 
> 
> > `pgbench -i -s 100` actually hang on my machine, because the
> > copy process ended up with waiting after `pg_uds_send_with_fd`
> > had
> 
> Hm, that had worked at some point...
> 
> 
> >     errno == EWOULDBLOCK || errno == EAGAIN
> > 
> > as well as the checkpointer process.
> 
> What do you mean with that latest sentence?
> 
> 
> > Looks like with the default
> > configuration and `max_wal_size=1GB` it writes more than reads to a
> > socket, and a buffer eventually becomes full.
> 
> That's intended to then wake up the checkpointer immediately, so it can
> absorb the requests.  So something isn't right yet.

Doesn't hang here, but it's way too slow. Reason for that is that I've
wrongly resolved a merge conflict. Attached is a fixup patch - does that
address the issue for you?

Greetings,

Andres Freund

Attachment

Re: Postgres, fsync, and OSs (specifically linux)

From
Dmitry Dolgov
Date:
> On 22 May 2018 at 18:47, Andres Freund <andres@anarazel.de> wrote:
> On 2018-05-22 08:57:18 -0700, Andres Freund wrote:
>> Hi,
>>
>>
>> On 2018-05-22 17:37:28 +0200, Dmitry Dolgov wrote:
>> > Thanks for the patch. Out of curiosity I tried to play with it a bit.
>>
>> Thanks.
>>
>>
>> > `pgbench -i -s 100` actually hang on my machine, because the
>> > copy process ended up with waiting after `pg_uds_send_with_fd`
>> > had
>>
>> Hm, that had worked at some point...
>>
>>
>> >     errno == EWOULDBLOCK || errno == EAGAIN
>> >
>> > as well as the checkpointer process.
>>
>> What do you mean with that latest sentence?

To investigate what's happening I attached with gdb to two processes, COPY
process from pgbench and checkpointer (since I assumed it may be involved).
Both were waiting in WaitLatchOrSocket right after SendFsyncRequest.

>> > Looks like with the default
>> > configuration and `max_wal_size=1GB` it writes more than reads to a
>> > socket, and a buffer eventually becomes full.
>>
>> That's intended to then wake up the checkpointer immediately, so it can
>> absorb the requests.  So something isn't right yet.
>
> Doesn't hang here, but it's way too slow.

Yep, in my case it was also getting slower, but eventually hang.

> Reason for that is that I've wrongly resolved a merge conflict. Attached is a
> fixup patch - does that address the issue for you?

Hm...is it a correct patch? I see the same committed in
8c3debbbf61892dabd8b6f3f8d55e600a7901f2b, so I can't really apply it.


Re: Postgres, fsync, and OSs (specifically linux)

From
Andres Freund
Date:
On 2018-05-22 20:54:46 +0200, Dmitry Dolgov wrote:
> > On 22 May 2018 at 18:47, Andres Freund <andres@anarazel.de> wrote:
> > On 2018-05-22 08:57:18 -0700, Andres Freund wrote:
> >> Hi,
> >>
> >>
> >> On 2018-05-22 17:37:28 +0200, Dmitry Dolgov wrote:
> >> > Thanks for the patch. Out of curiosity I tried to play with it a bit.
> >>
> >> Thanks.
> >>
> >>
> >> > `pgbench -i -s 100` actually hang on my machine, because the
> >> > copy process ended up with waiting after `pg_uds_send_with_fd`
> >> > had
> >>
> >> Hm, that had worked at some point...
> >>
> >>
> >> >     errno == EWOULDBLOCK || errno == EAGAIN
> >> >
> >> > as well as the checkpointer process.
> >>
> >> What do you mean with that latest sentence?
> 
> To investigate what's happening I attached with gdb to two processes, COPY
> process from pgbench and checkpointer (since I assumed it may be involved).
> Both were waiting in WaitLatchOrSocket right after SendFsyncRequest.

Huh? Checkpointer was in SendFsyncRequest()? Coudl you share the
backtrace?


> >> > Looks like with the default
> >> > configuration and `max_wal_size=1GB` it writes more than reads to a
> >> > socket, and a buffer eventually becomes full.
> >>
> >> That's intended to then wake up the checkpointer immediately, so it can
> >> absorb the requests.  So something isn't right yet.
> >
> > Doesn't hang here, but it's way too slow.
> 
> Yep, in my case it was also getting slower, but eventually hang.
> 
> > Reason for that is that I've wrongly resolved a merge conflict. Attached is a
> > fixup patch - does that address the issue for you?
> 
> Hm...is it a correct patch? I see the same committed in
> 8c3debbbf61892dabd8b6f3f8d55e600a7901f2b, so I can't really apply it.

Yea, sorry for that. Too many files in my patch directory... Right one
attached.

Greetings,

Andres Freund

Attachment

Re: Postgres, fsync, and OSs (specifically linux)

From
Dmitry Dolgov
Date:
> On 22 May 2018 at 20:59, Andres Freund <andres@anarazel.de> wrote:
> On 2018-05-22 20:54:46 +0200, Dmitry Dolgov wrote:
>> > On 22 May 2018 at 18:47, Andres Freund <andres@anarazel.de> wrote:
>> > On 2018-05-22 08:57:18 -0700, Andres Freund wrote:
>> >> Hi,
>> >>
>> >>
>> >> On 2018-05-22 17:37:28 +0200, Dmitry Dolgov wrote:
>> >> > Thanks for the patch. Out of curiosity I tried to play with it a bit.
>> >>
>> >> Thanks.
>> >>
>> >>
>> >> > `pgbench -i -s 100` actually hang on my machine, because the
>> >> > copy process ended up with waiting after `pg_uds_send_with_fd`
>> >> > had
>> >>
>> >> Hm, that had worked at some point...
>> >>
>> >>
>> >> >     errno == EWOULDBLOCK || errno == EAGAIN
>> >> >
>> >> > as well as the checkpointer process.
>> >>
>> >> What do you mean with that latest sentence?
>>
>> To investigate what's happening I attached with gdb to two processes, COPY
>> process from pgbench and checkpointer (since I assumed it may be involved).
>> Both were waiting in WaitLatchOrSocket right after SendFsyncRequest.
>
> Huh? Checkpointer was in SendFsyncRequest()? Coudl you share the
> backtrace?

Well, that's what I've got from gdb:

    #0  0x00007fae03fae9f3 in __epoll_wait_nocancel () at
../sysdeps/unix/syscall-template.S:84
    #1  0x000000000077a979 in WaitEventSetWaitBlock (nevents=1,
occurred_events=0x7ffe37529ec0, cur_timeout=-1, set=0x23cddf8) at
latch.c:1048
    #2  WaitEventSetWait (set=set@entry=0x23cddf8,
timeout=timeout@entry=-1,
occurred_events=occurred_events@entry=0x7ffe37529ec0,
nevents=nevents@entry=1, wait_event_info=wait_event_info@entry=0) at
latch.c:1000
    #3  0x000000000077ad08 in WaitLatchOrSocket
(latch=latch@entry=0x0, wakeEvents=wakeEvents@entry=4, sock=8,
timeout=timeout@entry=-1, wait_event_info=wait_event_info@entry=0) at
latch.c:385
    #4  0x00000000007152cb in SendFsyncRequest
(request=request@entry=0x7ffe37529f40, fd=fd@entry=-1) at
checkpointer.c:1345
    #5  0x0000000000716223 in AbsorbAllFsyncRequests () at checkpointer.c:1207
    #6  0x000000000079a5f0 in mdsync () at md.c:1339
    #7  0x000000000079c672 in smgrsync () at smgr.c:766
    #8  0x000000000076dd53 in CheckPointBuffers (flags=flags@entry=64)
at bufmgr.c:2581
    #9  0x000000000051c681 in CheckPointGuts
(checkPointRedo=722254352, flags=flags@entry=64) at xlog.c:9079
    #10 0x0000000000523c4a in CreateCheckPoint (flags=flags@entry=64)
at xlog.c:8863
    #11 0x0000000000715f41 in CheckpointerMain () at checkpointer.c:494
    #12 0x00000000005329f4 in AuxiliaryProcessMain (argc=argc@entry=2,
argv=argv@entry=0x7ffe3752a220) at bootstrap.c:451
    #13 0x0000000000720c28 in StartChildProcess
(type=type@entry=CheckpointerProcess) at postmaster.c:5340
    #14 0x0000000000721c23 in reaper (postgres_signal_arg=<optimized
out>) at postmaster.c:2875
    #15 <signal handler called>
    #16 0x00007fae03fa45b3 in __select_nocancel () at
../sysdeps/unix/syscall-template.S:84
    #17 0x0000000000722968 in ServerLoop () at postmaster.c:1679
    #18 0x0000000000723cde in PostmasterMain (argc=argc@entry=3,
argv=argv@entry=0x23a00e0) at postmaster.c:1388
    #19 0x000000000068979f in main (argc=3, argv=0x23a00e0) at main.c:228

>> >> > Looks like with the default
>> >> > configuration and `max_wal_size=1GB` it writes more than reads to a
>> >> > socket, and a buffer eventually becomes full.
>> >>
>> >> That's intended to then wake up the checkpointer immediately, so it can
>> >> absorb the requests.  So something isn't right yet.
>> >
>> > Doesn't hang here, but it's way too slow.
>>
>> Yep, in my case it was also getting slower, but eventually hang.
>>
>> > Reason for that is that I've wrongly resolved a merge conflict. Attached is a
>> > fixup patch - does that address the issue for you?
>>
>> Hm...is it a correct patch? I see the same committed in
>> 8c3debbbf61892dabd8b6f3f8d55e600a7901f2b, so I can't really apply it.
>
> Yea, sorry for that. Too many files in my patch directory... Right one
> attached.

Yes, this patch solves the problem, thanks.


Re: Postgres, fsync, and OSs (specifically linux)

From
Andres Freund
Date:
On 2018-05-22 21:58:06 +0200, Dmitry Dolgov wrote:
> > On 22 May 2018 at 20:59, Andres Freund <andres@anarazel.de> wrote:
> > On 2018-05-22 20:54:46 +0200, Dmitry Dolgov wrote:
> > Huh? Checkpointer was in SendFsyncRequest()? Coudl you share the
> > backtrace?
> 
> Well, that's what I've got from gdb:

>     #3  0x000000000077ad08 in WaitLatchOrSocket
> (latch=latch@entry=0x0, wakeEvents=wakeEvents@entry=4, sock=8,
> timeout=timeout@entry=-1, wait_event_info=wait_event_info@entry=0) at
> latch.c:385
>     #4  0x00000000007152cb in SendFsyncRequest
> (request=request@entry=0x7ffe37529f40, fd=fd@entry=-1) at
> checkpointer.c:1345
>     #5  0x0000000000716223 in AbsorbAllFsyncRequests () at checkpointer.c:1207

Oh, I see. That makes sense. So it's possible to self-deadlock
here. Should be easy to fix...  Thanks for finding that one.


> Yes, this patch solves the problem, thanks.

Just avoids it, I'm afraid... It probably can't be hit easily, but the
issue is there...

- Andres


Re: Postgres, fsync, and OSs (specifically linux)

From
Craig Ringer
Date:
On 21 May 2018 at 15:50, Craig Ringer <craig@2ndquadrant.com> wrote:
On 21 May 2018 at 12:57, Craig Ringer <craig@2ndquadrant.com> wrote:
On 18 May 2018 at 00:44, Andres Freund <andres@anarazel.de> wrote:
Hi,

On 2018-05-10 09:50:03 +0800, Craig Ringer wrote:
>       while ((src = (RewriteMappingFile *) hash_seq_search(&seq_status)) != NULL)
>       {
>               if (FileSync(src->vfd, WAIT_EVENT_LOGICAL_REWRITE_SYNC) != 0)
> -                     ereport(ERROR,
> +                     ereport(PANIC,
>                                       (errcode_for_file_access(),
>                                        errmsg("could not fsync file \"%s\": %m", src->path)));

To me this (and the other callers) doesn't quite look right. First, I
think we should probably be a bit more restrictive about when PANIC
out. It seems like we should PANIC on ENOSPC and EIO, but possibly not
others.  Secondly, I think we should centralize the error handling. It
seems likely that we'll acrue some platform specific workarounds, and I
don't want to copy that knowledge everywhere.

Also, don't we need the same on close()?


Yes, we do, and that expands the scope a bit.

I agree with Robert that some sort of filter/macro is wise, though naming it clearly will be tricky.

I'll have a look.


On the queue for tomorrow. 


Hi all.

I've revised the fsync patch with the cleanups discussed and gone through the close() calls.

AFAICS either socket closes, temp file closes, or (for WAL) already PANIC on close.  It's mainly fd.c that needs amendment. Which I've done per the attached revised patch.


--
 Craig Ringer                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services
Attachment

Re: Postgres, fsync, and OSs (specifically linux)

From
Thomas Munro
Date:
On Wed, May 23, 2018 at 8:02 AM, Andres Freund <andres@anarazel.de> wrote:
> [patches]

Hi Andres,

Obviously there is more work to be done here but the basic idea in
your clone-fd-checkpointer branch as of today seems OK to me.  I think
Craig and I both had similar ideas (sending file descriptors that have
an old enough f_wb_err) but we thought rlimit management was going to
be hard (shared memory counters + deduplication, bleugh).  You made it
look easy.  Nice work.

First, let me describe in my own words what's going on, mostly to make
sure I really understand this:

1.  We adopt a new fd lifecycle that is carefully tailored to avoid
error loss on Linux, and can't hurt on other OSes.  By sending the
file descriptor used for every write to the checkpointer, we guarantee
that (1) the inode stays pinned (the file is "in flight" in the
socket, even if the sender closes it before the checkpointer receives
it) so Linux won't be tempted to throw away the precious information
in i_mapping->wb_err, and (2) the checkpointer finishes up with a file
descriptor that points to the very same "struct file" with the
f_wb_err value that was originally sampled before the write, by the
sender.  So we can't miss any write-back errors.  Wahey!  However,
errors are still reported only once, so we probably need to panic.

Hmm...  Is there any way that the *sender* could finish up in
file_check_and_advance_wb_err() for the same struct file, before the
checkpointer manages to call fsync() on its dup'd fd?  I don't
immediately see how (it looks like you have to call one of the various
sync functions to reach that, and your patch removes the fallback
just-call-FileSync()-myself code from register_dirty_segment()).  I
guess it would be bad if, say, close() were to do that in some future
Linux release because then we'd have no race-free way to tell the
checkpointer that the file is borked before it runs fsync() and
potentially gets an OK response and reports a successful checkpoint
(even if we panicked, with sufficiently bad timing it might manage to
report a successful checkpoint).

2.  In order to make sure that we don't exceed our soft limit on the
number of file descriptors per process, you use a simple backend-local
counter in the checkpointer, on the theory that we don't care about
fds (or, technically, the files they point to) waiting in the socket
buffer, we care only about how many the checkpointer has actually
received but not yet closed.  As long as we make sure there is space
for at least one more before we read one message, everything should be
fine.  Good and simple.

One reason I thought this would be harder is because I had no model of
how RLIMIT_NOFILE would apply when you start flinging fds around
between processes (considering there can be moments when neither end
has the file open), so I feared the worst and thought we would need to
maintain a counter in shared memory and have senders and receiver
cooperate to respect it.  My intuition that this was murky and
required pessimism wasn't too far from the truth actually: apparently
the kernel didn't do a great job at accounting for that until a
patch[1] landed for CVE-2013-4312.

The behaviour in older releases is super lax, so no problem there.
The behaviour from 4.9 onward (or is it 4.4.1?) is that you get a
separate per-user RLIMIT_NOFILE allowance for in-flight fds.  So
effectively the sender doesn't have to worry about about fds it has
sent but closed and the receiver doesn't have to care about fds it
hasn't received yet, so your counting scheme seems OK.  As for
exceeding RLIMIT_NOFILE with in-flight fds, it's at least bounded by
the fact that the socket would block/EWOULDBLOCK if the receiver isn't
draining it fast enough and can only hold a small and finite amount of
data and thus file descriptors, so we can probably just ignore that.
If you did manage to exceed it, I think you'd find out about that with
ETOOMANYREFS at send time (curiously absent from the sendmsg() man
page, but there in black and white in the source for
unix_attach_fds()), and then you'd just raise an error (currently
FATAL in your patch).  I have no idea how the rlimit for SCM-ified
files works on other Unixoid systems though.

Some actual code feedback:

+                       if (entry->syncfds[forknum][segno] == -1)
+                       {
+                               char *path = mdpath(entry->rnode,
forknum, segno);
+                               open_fsync_queue_files++;
+                               /* caller must have reserved entry */
+                               entry->syncfds[forknum][segno] =
+                                       FileOpenForFd(fd, path);
+                               pfree(path);
+                       }
+                       else
+                       {
+                               /*
+                                * File is already open. Have to keep
the older fd, errors
+                                * might only be reported to it, thus
close the one we just
+                                * got.
+                                *
+                                * XXX: check for errrors.
+                                */
+                               close(fd);
+                       }

Wait... it's not guaranteed to be older in open() time, is it?  It's
just older in sendmsg() time.  Those might be different:

A: open("base/42/1234")
A: write()
kernel: inode->i_mapping->wb_err++
B: open("base/42/1234")
B: write()
B: sendmsg()
A: sendmsg()
C: recvmsg() /* clone B's fd */
C: recvmsg() /* clone A's fd, throw it away because we already have B's */
C: fsync()

I think that would eat an error under the 4.13 code.  I think it might
be OK under the new 4.17 code, because the new "must report it to at
least one caller" thing would save the day.  So now I'm wondering if
that's getting backported to the 4.14 long term kernel.

Aha, yes it has been already[2].

So... if you don't have to worry about kernels without that patch, I
suspect the exact ordering doesn't matter anymore, as long as
*someone* held the file open at all times beween write and fsync() to
keep the inode around, which this patch achieves.  (And of course no
other process sets the ERRSEQ_SEEN flag, for example some other kind
of backend or random other program that opened our data file and
called one of the sync syscalls, or any future syscalls that start
calling file_check_and_advance_wb_err()).

+       /*
+        * FIXME: do DuplicateHandle dance for windows - can that work
+        * trivially?
+        */

I don't know, I guess something like CreatePipe() and then
write_duplicate_handle()?  And some magic (maybe our own LWLock) to
allow atomic messages?

A more interesting question is: how will you cap the number file
handles you send through that pipe?  On that OS you call
DuplicateHandle() to fling handles into another process's handle table
directly.  Then you send the handle number as plain old data to the
other process via carrier pigeon, smoke signals, a pipe etc.  That's
interesting because the handle allocation is asynchronous from the
point of view of the receiver.  Unlike the Unix case where the
receiver can count handles and make sure there is space for one more
before it reads a potentially-SCM-containing message, here the
*senders* will somehow need to make sure they don't create too many in
the receiving process.  I guess that would involve a shared counter,
and a strategy for what to do when the number is too high (probably
just wait).

Hmm.  I wonder if that would be a safer approach on all operating systems.

+       if (socketpair(AF_UNIX, SOCK_STREAM | SOCK_CLOEXEC, 0, fsync_fds) < 0)
...
+       size = sendmsg(sock, &msg, 0);

Here you are relying on atomic sending and receiving of whole messages
over a stream socket.  For example, in Linux's unix_stream_sendmsg()
it's going to be chopped into buffers of size (sk->sk_sndbuf >> 1) -
64 (I have no clue how big that is) that are appended one at a time to
the receiving socket's queue, with no locking in between, so a
concurrently send messages can be interleaved with yours.  How big is
big enough for that to happen?  There doesn't seem to be an equivalent
of PIPE_BUF for Unix domain sockets.  These tiny messages are almost
certainly safe, but I wonder if we should be using SOCK_SEQPACKET
instead of SOCK_STREAM?

Might be a less theoretical problem if we switch to variable sized
messages containing file paths as you once contemplated in an off-list
chat.

Presumably for EXEC_BACKEND we'll need to open
PGDATA/something/something/socket or similar.

+                                       if (returnCode < 0)
+                                       {
+                                               /* XXX: decide on policy */
+
bms_add_member(entry->requests[forknum], segno);

Obviously this is pseudocode (doesn't even keep the return value), but
just BTW, I think that if we decide not to PANIC unconditionally on
any kind of fsync() failure, we definitely can't use bms_add_member()
here (it might fail to allocate, and then we forget the segment, raise
and error and won't try again).  It's got to be PANIC or no-fail code
(like the patch I proposed in another thread).

+SendFsyncRequest(CheckpointerRequest *request, int fd)
...
+                        * Don't think short reads will ever happen in realistic
...
+                       ereport(FATAL, (errmsg("could not receive
fsync request: %m")));

Short *writes*, could not *send*.

+                * back, as that'd lead to loosing error reporting guarantees on

s/loosing/losing/

[1] https://github.com/torvalds/linux/commit/712f4aad406bb1ed67f3f98d04c044191f0ff593
[2]
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git/commit/lib/errseq.c?h=linux-4.14.y&id=0799a0ea96e4923f52f85fe315b62e9176a3319c

-- 
Thomas Munro
http://www.enterprisedb.com


Re: Postgres, fsync, and OSs (specifically linux)

From
Robert Haas
Date:
On Tue, May 29, 2018 at 4:53 AM, Craig Ringer <craig@2ndquadrant.com> wrote:
> I've revised the fsync patch with the cleanups discussed and gone through
> the close() calls.
>
> AFAICS either socket closes, temp file closes, or (for WAL) already PANIC on
> close.  It's mainly fd.c that needs amendment. Which I've done per the
> attached revised patch.

I think we should have a separate thread for this patch vs. Andres's
patch to do magic things with the checkpointer and file-descriptor
forwarding.  Meanwhile, here's some review.

1. No longer applies cleanly.

2. I don't like promote_ioerr_to_panic() very much, partly because the
same pattern gets repeated over and over, and partly because it would
be awkwardly-named if we discovered that another 2 or 3 errors needed
similar handling (or some other variant handling).  I suggest instead
having a function like report_critical_fsync_failure(char *path) that
does something like this:

    int elevel = ERROR;
    if (errno == EIO)
        elevel = PANIC;
    ereport(elevel,
                (errcode_for_file_access(),
                 errmsg("could not fsync file \"%s\": %m", path);

And similarly I'd add report_critical_close_failure.  In some cases,
this would remove wording variations (e.g. in twophase.c) but I think
that's fine, and maybe an improvement, as discussed on another recent
thread.

3. slru.c calls pg_fsync() but isn't changed by the patch.  That looks wrong.

4. The comment changes in snapbuild.c interact with the TODO that
immediately follows.  I think more adjustment is needed here.

5. It seems odd that you adjusted the comment for
pg_fsync_no_writethrough() but not pg_fsync_writethrough() or
pg_fsync().  Either pg_fsync_writethrough() doesn't have the same
problem, in which case, awesome, but let's add a comment, or it does,
in which case it should refer to the other one.  And I think
pg_fsync() itself needs a comment saying that every caller must be
careful to use promote_ioerr_to_panic() or
report_critical_fsync_failure() or whatever we end up calling it
unless the fsync is not critical for data integrity.

6. In md.c, there's a stray blank line added.  But more importantly,
the code just above that looks like this:

                     if (!FILE_POSSIBLY_DELETED(errno) ||
                         failures > 0)
-                        ereport(ERROR,
+                        ereport(promote_ioerr_to_panic(ERROR),
                                 (errcode_for_file_access(),
                                  errmsg("could not fsync file \"%s\": %m",
                                         path)));
                     else
                         ereport(DEBUG1,
                                 (errcode_for_file_access(),
                                  errmsg("could not fsync file \"%s\"
but retrying: %m",
                                         path)));

I might be all wet here, but it seems like if we enter the bottom
branch, we still need the promote-to-panic behavior.

7. The comment adjustment for SyncDataDirectory mentions an
"important" fact about fsync behavior, but then doesn't seem to change
any logic on that basis.  I think in general a number of these
comments need a little more thought, but in this particular case, I
think we also need to consider what the behavior should be (and the
comment should reflect our considered judgement on that point, and the
implementation should match).

8. Andres suggested to me off-list that we should have a GUC to
disable the promote-to-panic behavior in case it turns out to be a
show-stopper for some user.  I think that's probably a good idea.
Adding many new ways to PANIC in a minor release without providing any
way to go back to the old behavior sounds unfriendly.  Surely, anyone
who suffers much from this has really serious other problems anyway,
but all the same I think we should provide an escape hatch.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company


Re: Postgres, fsync, and OSs (specifically linux)

From
Thomas Munro
Date:
On Thu, Jul 19, 2018 at 7:23 AM, Robert Haas <robertmhaas@gmail.com> wrote:
> 2. I don't like promote_ioerr_to_panic() very much, partly because the
> same pattern gets repeated over and over, and partly because it would
> be awkwardly-named if we discovered that another 2 or 3 errors needed
> similar handling (or some other variant handling).  I suggest instead
> having a function like report_critical_fsync_failure(char *path) that
> ...

Note that if we don't cover *all* errno values, or ...

> 8. Andres suggested to me off-list that we should have a GUC to
> disable the promote-to-panic behavior in case it turns out to be a
> show-stopper for some user.

... we let the user turn this off, then we also have to fix this:

https://www.postgresql.org/message-id/flat/87y3i1ia4w.fsf@news-spur.riddles.org.uk

-- 
Thomas Munro
http://www.enterprisedb.com


Re: Postgres, fsync, and OSs (specifically linux)

From
Thomas Munro
Date:
On Thu, Jun 14, 2018 at 5:30 PM, Thomas Munro
<thomas.munro@enterprisedb.com> wrote:
> On Wed, May 23, 2018 at 8:02 AM, Andres Freund <andres@anarazel.de> wrote:
>> [patches]
>
> A more interesting question is: how will you cap the number file
> handles you send through that pipe?  On that OS you call
> DuplicateHandle() to fling handles into another process's handle table
> directly.  Then you send the handle number as plain old data to the
> other process via carrier pigeon, smoke signals, a pipe etc.  That's
> interesting because the handle allocation is asynchronous from the
> point of view of the receiver.  Unlike the Unix case where the
> receiver can count handles and make sure there is space for one more
> before it reads a potentially-SCM-containing message, here the
> *senders* will somehow need to make sure they don't create too many in
> the receiving process.  I guess that would involve a shared counter,
> and a strategy for what to do when the number is too high (probably
> just wait).
>
> Hmm.  I wonder if that would be a safer approach on all operating systems.

As a way of poking this thread, here are some more thoughts.  Buffer
stealing currently look something like this:

Evicting backend:
  lseek(fd)
  write(fd)
  ...enqueue-fsync-request via shm...

Checkpointer:
  ...push into hash table...

With the patch it presumably looks something like this:

Evicting backend:
  lseek(fd)
  write(fd)
  sendmsg(fsync_socket) /* passes fd */

Checkpointer:
  recvmsg(fsync_socket) /* gets a copy of fd */
  ...push into hash table...
  close(fd) /* for all but the first one received for the same file */

That takes us from 2 syscalls to 5 per evicted buffer.  I suppose it's
possible that on some operating systems that might hurt a bit, given
that it's happening at the granularity of 1GB data files that could
have a lot of backends working in them concurrently.  I have no idea
if it's really a problem on any particular OS.  Admittedly on Linux
it's probably just a bunch of fast atomic ops and RCU stuff...
probably only the existing write() actually takes the inode lock or
anything that heavy, and that's probably lost in the noise in an
evict-heavy workload.  I don't know, I guess it's probably not a
problem, but I thought I'd mention that.

Contention on the new fsync socket doesn't seem to be a new problem
per se since it replaces a contention point we already had:
CheckpointerCommLock.  If that was acceptable today then perhaps that
indicates that any in-kernel contention created by the new syscalls is
also OK.

My feeling so far is that I'd probably go for sender-collapses model
(and it might even be necessary on Windows?) if doing this as a new
feature, but I fully understand your desire to do it in a much simpler
way that could be back-patched more easily.  I'm just slightly
concerned about the unintended consequence risk that comes with
exercising an operating system feature that not all operating system
authors probably intended to be used at high frequency.  Nothing that
can't be assuaged by testing.

  * the queue is full and contains no duplicate entries.  In that case, we
  * let the backend know by returning false.
  */
-bool
-ForwardFsyncRequest(RelFileNode rnode, ForkNumber forknum, BlockNumber segno)
+void
+ForwardFsyncRequest(RelFileNode rnode, ForkNumber forknum, BlockNumber segno,
+                                       File file)

Comment out of date.

-- 
Thomas Munro
http://www.enterprisedb.com


Re: Postgres, fsync, and OSs (specifically linux)

From
Thomas Munro
Date:
On Sun, Jul 29, 2018 at 6:14 PM, Thomas Munro
<thomas.munro@enterprisedb.com> wrote:
> As a way of poking this thread, here are some more thoughts.

I am keen to move this forward, not only because it is something we
need to get fixed, but also because I have some other pending patches
in this area and I want this sorted out first.

Here are some small fix-up patches for Andres's patchset:

1.  Use FD_CLOEXEC instead of the non-portable Linuxism SOCK_CLOEXEC.

2.  Fix the self-deadlock hazard reported by Dmitry Dolgov.  Instead
of the checkpoint trying to send itself a CKPT_REQUEST_SYN message
through the socket (whose buffer may be full), I included the
ckpt_started counter in all messages.  When AbsorbAllFsyncRequests()
drains the socket, it stops at messages with the current ckpt_started
value.

3.  Handle postmaster death while waiting.

4.  I discovered that macOS would occasionally return EMSGSIZE for
sendmsg(), but treating that just like EAGAIN seems to work the next
time around.  I couldn't make that happen on FreeBSD (I mention that
because the implementation is somehow related).  So handle that weird
case on macOS only for now.

Testing on other Unixoid systems would be useful.  The case that
produced occasional EMSGSIZE on macOS was: shared_buffers=1MB,
max_files_per_process=32, installcheck-parallel.  Based on man pages
that seems to imply an error in the client code but I don't see it.

(I also tried to use SOCK_SEQPACKET instead of SOCK_STREAM, but it's
not supported on macOS.  I also tried to use SOCK_DGRAM, but that
produced occasional ENOBUFS errors and retrying didn't immediately
succeed leading to busy syscall churn.  This is all rather
unsatisfying, since SOCK_STREAM is not guaranteed by any standard to
be atomic, and we're writing messages from many backends into the
socket so we're assuming atomicity.  I don't have a better idea that
is portable.)

There are a couple of FIXMEs remaining, and I am aware of three more problems:

* Andres mentioned to me off-list that there may be a deadlock risk
where the checkpointer gets stuck waiting for an IO lock.  I'm going
to look into that.
* Windows.  Patch soon.
* The ordering problem that I mentioned earlier: the patchset wants to
keep the *oldest* fd, but it's really the oldest it has received.  An
idea Andres and I discussed is to use a shared atomic counter to
assign a number to all file descriptors just before their first write,
and send that along with it to the checkpointer.  Patch soon.

-- 
Thomas Munro
http://www.enterprisedb.com

Attachment

Re: Postgres, fsync, and OSs (specifically linux)

From
Asim R P
Date:
I was looking at the commitfest entry for feature
(https://commitfest.postgresql.org/19/1639/) for the most recent list
of patches to try out.  The list doesn't look correct/complete.  Can
someone please check?

Asim


Re: Postgres, fsync, and OSs (specifically linux)

From
Thomas Munro
Date:
On Wed, Aug 15, 2018 at 11:08 AM, Asim R P <apraveen@pivotal.io> wrote:
> I was looking at the commitfest entry for feature
> (https://commitfest.postgresql.org/19/1639/) for the most recent list
> of patches to try out.  The list doesn't look correct/complete.  Can
> someone please check?

Hi Asim,

This thread is a bit tangled up.  There are two related patchsets in it:

1.  Craig Ringer's PANIC-on-EIO patch set, to cope with the fact that
Linux throws away buffers and errors after reporting an error, so the
checkpointer shouldn't retry as it does today.  The latest is here:

https://www.postgresql.org/message-id/CAMsr%2BYFPeKVaQ57PwHqmRNjPCPABsdbV%3DL85he2dVBcr6yS1mA%40mail.gmail.com

2.  Andres Freund's fd-sending fsync queue, to cope with the fact that
some versions of Linux only report writeback errors that occurred
after you opened the file, and all versions of Linux and some other
operating systems might forget about writeback errors while no one has
it open.

Here is the original patchset:

https://www.postgresql.org/message-id/20180522010823.z5bdq7wnlsna5qoo%40alap3.anarazel.de

Here is a fix-up you need:

https://www.postgresql.org/message-id/20180522185951.5sdudzl46spktyyz%40alap3.anarazel.de

Here are some more fix-up patches that I propose:

https://www.postgresql.org/message-id/CAEepm%3D2WSPP03-20XHpxohSd2UyG_dvw5zWS1v7Eas8Rd%3D5e4A%40mail.gmail.com

I will soon post some more fix-up patches that add EXEC_BACKEND
support, Windows support, and a counting scheme to fix the timing
issue that I mentioned in my first review.  I will probably squash it
all down to a tidy patch-set after that.

-- 
Thomas Munro
http://www.enterprisedb.com


Re: Postgres, fsync, and OSs (specifically linux)

From
Craig Ringer
Date:
On 15 August 2018 at 07:32, Thomas Munro <thomas.munro@enterprisedb.com> wrote:
On Wed, Aug 15, 2018 at 11:08 AM, Asim R P <apraveen@pivotal.io> wrote:
> I was looking at the commitfest entry for feature
> (https://commitfest.postgresql.org/19/1639/) for the most recent list
> of patches to try out.  The list doesn't look correct/complete.  Can
> someone please check?

Hi Asim,

This thread is a bit tangled up.  There are two related patchsets in it:

1.  Craig Ringer's PANIC-on-EIO patch set, to cope with the fact that
Linux throws away buffers and errors after reporting an error, so the
checkpointer shouldn't retry as it does today.  The latest is here:

https://www.postgresql.org/message-id/CAMsr%2BYFPeKVaQ57PwHqmRNjPCPABsdbV%3DL85he2dVBcr6yS1mA%40mail.gmail.com

2.  Andres Freund's fd-sending fsync queue, to cope with the fact that
some versions of Linux only report writeback errors that occurred
after you opened the file, and all versions of Linux and some other
operating systems might forget about writeback errors while no one has
it open.

Here is the original patchset:

https://www.postgresql.org/message-id/20180522010823.z5bdq7wnlsna5qoo%40alap3.anarazel.de

Here is a fix-up you need:

https://www.postgresql.org/message-id/20180522185951.5sdudzl46spktyyz%40alap3.anarazel.de

Here are some more fix-up patches that I propose:

https://www.postgresql.org/message-id/CAEepm%3D2WSPP03-20XHpxohSd2UyG_dvw5zWS1v7Eas8Rd%3D5e4A%40mail.gmail.com

I will soon post some more fix-up patches that add EXEC_BACKEND
support, Windows support, and a counting scheme to fix the timing
issue that I mentioned in my first review.  I will probably squash it
all down to a tidy patch-set after that.


Thanks very much Tomas. 

I've had to back off from this a bit after posting my initial panic-for-safety patch, as the changes Andres proposed are a bit out of my current depth and time capacity.

I still think the panic patch is needed and appropriate, but agree it's not *sufficient*.

Re: Postgres, fsync, and OSs (specifically linux)

From
Thomas Munro
Date:
On Thu, Aug 30, 2018 at 2:44 PM Craig Ringer <craig@2ndquadrant.com> wrote:
> On 15 August 2018 at 07:32, Thomas Munro <thomas.munro@enterprisedb.com> wrote:
>> I will soon post some more fix-up patches that add EXEC_BACKEND
>> support, Windows support, and a counting scheme to fix the timing
>> issue that I mentioned in my first review.  I will probably squash it
>> all down to a tidy patch-set after that.

I went down a bit of a rabbit hole with the Windows support for
Andres's patch set.  I have something that works as far as I can tell,
but my Windows environment consists of throwing things at Appveyor and
seeing what sticks, so I'm hoping that someone with a real Windows
system and knowledge will be able to comment.

New patches in this WIP patch set:

0012: Fix for EXEC_BACKEND.

0013: Windows.  This involved teaching latch.c to deal with Windows
asynchronous IO events, since you can't wait for pipe readiness via
WSAEventSelect.  Pipes and sockets exist in different dimensions on
Windows, and there are no "Unix" domain sockets (well, there are but
they aren't usable yet[1]).  An alternative would be to use TCP
sockets for this, and then the code would look more like the Unix
code, but that seems a bit strange.  Note that the Windows version
doesn't actually hand off file handles like the Unix code (it could
fairly easily, but there is no reason to think that would actually be
useful on that platform).  I may be way off here...

The 0013 patch also fixes a mistake in the 0010 patch: it is not
appropriate to call CFI() while waiting to notify the checkpointer of
a dirty segment, because then ^C could cause the following checkpoint
not to flush dirty data.  SendFsyncRequest() is essentially blocking,
except that it uses non-blocking IO so that it multiplex postmaster
death detection.

0014: Fix the ordering race condition mentioned upthread[2].  All
files are assigned an increasing sequence number after [re]opening (ie
before their first write), so that the checkpointer process can track
the fd that must have the oldest Linux f_wb_err that could be relevant
for writes done by PostgreSQL.

The other patches in this tarball are all as posted already, but are
now rebased and assembled in one place.  Also pushed to
https://github.com/macdice/postgres/tree/fsyncgate .

Thoughts?

[1] https://blogs.msdn.microsoft.com/commandline/2017/12/19/af_unix-comes-to-windows/
[2] https://www.postgresql.org/message-id/CAEepm%3D04ZCG_8N3m61kXZP-7Ecr02HUNNG-QsAhwyFLim4su2g%40mail.gmail.com

-- 
Thomas Munro
http://www.enterprisedb.com

Attachment

Re: Postgres, fsync, and OSs (specifically linux)

From
Thomas Munro
Date:
On Fri, Sep 28, 2018 at 9:37 PM Thomas Munro
<thomas.munro@enterprisedb.com> wrote:
> The 0013 patch also fixes a mistake in the 0010 patch: it is not
> appropriate to call CFI() while waiting to notify the checkpointer of
> a dirty segment, because then ^C could cause the following checkpoint
> not to flush dirty data.

(Though of course it wouldn't actually do that due to an LWLock being
held, but still, I removed the CFI because it was at best misleading).

-- 
Thomas Munro
http://www.enterprisedb.com


Re: Postgres, fsync, and OSs (specifically linux)

From
Thomas Munro
Date:
On Fri, Sep 28, 2018 at 9:37 PM Thomas Munro
<thomas.munro@enterprisedb.com> wrote:
> The other patches in this tarball are all as posted already, but are
> now rebased and assembled in one place.  Also pushed to
> https://github.com/macdice/postgres/tree/fsyncgate .

Here is a new version that fixes an assertion failure during crash
recovery, revealed by cfbot.  I also took the liberty of squashing the
patch stack into one and writing a new commit message, except the
Windows part which seems worth keeping separate until we agree it's
the right way forward.

-- 
Thomas Munro
http://www.enterprisedb.com

Attachment

Re: Postgres, fsync, and OSs (specifically linux)

From
Thomas Munro
Date:
Hello hackers,

Let's try to get this issue resolved.  Here is my position on the
course of action we should take in back-branches:

1.  I am -1 on back-patching the fd-transfer code.  It's a significant
change, and even when sufficiently debugged (I don't think it's there
yet), we have no idea what will happen on all the kernels we support
under extreme workloads.  IMHO there is no way we can spring this on
users in a point release.

2.  I am +1 on back-patching Craig's PANIC-on-failure logic.  Doing
nothing is not an option I like.  I have some feedback and changes to
propose though; see attached.

Responses to a review from Robert:

On Thu, Jul 19, 2018 at 7:23 AM Robert Haas <robertmhaas@gmail.com> wrote:
> 2. I don't like promote_ioerr_to_panic() very much, partly because the
> same pattern gets repeated over and over, and partly because it would
> be awkwardly-named if we discovered that another 2 or 3 errors needed
> similar handling (or some other variant handling).  I suggest instead
> having a function like report_critical_fsync_failure(char *path) that
> does something like this:
>
>     int elevel = ERROR;
>     if (errno == EIO)
>         elevel = PANIC;
>     ereport(elevel,
>                 (errcode_for_file_access(),
>                  errmsg("could not fsync file \"%s\": %m", path);
>
> And similarly I'd add report_critical_close_failure.  In some cases,
> this would remove wording variations (e.g. in twophase.c) but I think
> that's fine, and maybe an improvement, as discussed on another recent
> thread.

I changed it to look like data_sync_elevel(ERROR) and made it treat
all errnos the same.  ENOSPC, EIO, EWOK, EIEIO, it makes no difference
to the level of faith I have that my data still exists.

> 3. slru.c calls pg_fsync() but isn't changed by the patch.  That looks wrong.

Fixed.

> 4. The comment changes in snapbuild.c interact with the TODO that
> immediately follows.  I think more adjustment is needed here.

I don't understand this.

> 5. It seems odd that you adjusted the comment for
> pg_fsync_no_writethrough() but not pg_fsync_writethrough() or
> pg_fsync().  Either pg_fsync_writethrough() doesn't have the same
> problem, in which case, awesome, but let's add a comment, or it does,
> in which case it should refer to the other one.  And I think
> pg_fsync() itself needs a comment saying that every caller must be
> careful to use promote_ioerr_to_panic() or
> report_critical_fsync_failure() or whatever we end up calling it
> unless the fsync is not critical for data integrity.

I removed these comments and many others; I don't see the point in
scattering descriptions of this problem and references to specific
versions of Linux and -hackers archive links all over the place.  I
added a comment in one place, and also added some user documentation
of the problem.

> 6. In md.c, there's a stray blank line added.  But more importantly,
> the code just above that looks like this:
>
>                      if (!FILE_POSSIBLY_DELETED(errno) ||
>                          failures > 0)
> -                        ereport(ERROR,
> +                        ereport(promote_ioerr_to_panic(ERROR),
>                                  (errcode_for_file_access(),
>                                   errmsg("could not fsync file \"%s\": %m",
>                                          path)));
>                      else
>                          ereport(DEBUG1,
>                                  (errcode_for_file_access(),
>                                   errmsg("could not fsync file \"%s\"
> but retrying: %m",
>                                          path)));
>
> I might be all wet here, but it seems like if we enter the bottom
> branch, we still need the promote-to-panic behavior.

That case only is only reached if FILE_POSSIBLY_DELETED() on the first
time through the loop, and it detects an errno value not actually from
fsync().  It's from FileSync(), when it tries to reopen a virtual fd
and gets ENOENT, before calling fsync().  Code further down then
absorbs incoming requests before checking if that was expected,
closing a race.  The comments could make that clearer, admittedly.

> 7. The comment adjustment for SyncDataDirectory mentions an
> "important" fact about fsync behavior, but then doesn't seem to change
> any logic on that basis.  I think in general a number of these
> comments need a little more thought, but in this particular case, I
> think we also need to consider what the behavior should be (and the
> comment should reflect our considered judgement on that point, and the
> implementation should match).

I updated the comment.  I don't think this is too relevant to the
fsync() failure case, because we'll be rewriting all changes from the
WAL again during recovery; I think this function is mostly useful for
switching from fsync = off to fsync = on and restarting, not coping
with previous fsync() failures by retrying (which we know to be
useless anyway).  Someone could argue that if you restarted after
changing fsync from off to on, then this may be the first time you
learn that write-back failed, and then you're somewhat screwed whether
we panic or not, but I don't see any solution to that.  Don't run
databases with fsync = off.

> 8. Andres suggested to me off-list that we should have a GUC to
> disable the promote-to-panic behavior in case it turns out to be a
> show-stopper for some user.  I think that's probably a good idea.
> Adding many new ways to PANIC in a minor release without providing any
> way to go back to the old behavior sounds unfriendly.  Surely, anyone
> who suffers much from this has really serious other problems anyway,
> but all the same I think we should provide an escape hatch.

+1.  See the new GUC data_sync_retry, defaulting to false.  If set to
true, we also need to fix the problem reported in [1], so here's the
patch for that too.

Other comments:

I don't see why sync_file_range(SYNC_FILE_RANGE_WRITE) should get a
pass here.  Inspection of some version of the kernel might tell us it
can't advance the error counter and report failure, but what do we
gain by relying on that?  Changed.

FD_DELETE_AT_CLOSE is not a good way to detect temporary files in
recent versions, as it doesn't detect the kind of shared temporary
files used by Parallel Hash; FD_TEMP_FILE_LIMIT is a better way.
Changed.  (We could also just not bother exempting temporary files?)

I plan to continue working on the fd-transfer system as part of a
larger sync queue redesign project for 12.  If we can get an agreement
that we can't possibly back-patch the fd-transfer logic, then we can
move all future discussion of that topic over to the other thread[2],
and this thread can be about consensus to back-patch the PANIC patch.
Thoughts?

[1] https://www.postgresql.org/message-id/flat/87y3i1ia4w.fsf@news-spur.riddles.org.uk
[2] https://www.postgresql.org/message-id/flat/CAEepm=2gTANm=e3ARnJT=n0h8hf88wqmaZxk0JYkxw+b21fNrw@mail.gmail.com

Attachment

Re: Postgres, fsync, and OSs (specifically linux)

From
Craig Ringer
Date:
On Fri, 19 Oct 2018 at 07:27, Thomas Munro <thomas.munro@enterprisedb.com> wrote:

2.  I am +1 on back-patching Craig's PANIC-on-failure logic.  Doing
nothing is not an option I like.  I have some feedback and changes to
propose though; see attached.

Thanks very much for the work on reviewing and revising this.
 
I don't see why sync_file_range(SYNC_FILE_RANGE_WRITE) should get a
pass here.  Inspection of some version of the kernel might tell us it
can't advance the error counter and report failure, but what do we
gain by relying on that?  Changed.

I was sure it made sense at the time, but I can't explain that decision now, and it looks like we should treat it as a failure.

--
 Craig Ringer                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

Re: Postgres, fsync, and OSs (specifically linux)

From
Thomas Munro
Date:
On Fri, Oct 19, 2018 at 6:42 PM Craig Ringer <craig@2ndquadrant.com> wrote:
> On Fri, 19 Oct 2018 at 07:27, Thomas Munro <thomas.munro@enterprisedb.com> wrote:
>> 2.  I am +1 on back-patching Craig's PANIC-on-failure logic.  Doing
>> nothing is not an option I like.  I have some feedback and changes to
>> propose though; see attached.
>
> Thanks very much for the work on reviewing and revising this.

My plan is do a round of testing and review of this stuff next week
once the dust is settled on the current minor releases (including
fixing a few typos I just spotted and some word-smithing).  All going
well, I will then push the resulting patches to master and all
supported stable branches, unless other reviews or objections appear.
At some point not too far down the track I hope to be ready to
consider committing that other patch that will completely change all
of this code in the master branch, but in any case Craig's patch will
get almost a full minor release cycle to sit in the stable branches
before release.

-- 
Thomas Munro
http://www.enterprisedb.com


Re: Postgres, fsync, and OSs (specifically linux)

From
Robert Haas
Date:
On Wed, Nov 7, 2018 at 9:41 PM Thomas Munro
<thomas.munro@enterprisedb.com> wrote:
> My plan is do a round of testing and review of this stuff next week
> once the dust is settled on the current minor releases (including
> fixing a few typos I just spotted and some word-smithing).  All going
> well, I will then push the resulting patches to master and all
> supported stable branches, unless other reviews or objections appear.
> At some point not too far down the track I hope to be ready to
> consider committing that other patch that will completely change all
> of this code in the master branch, but in any case Craig's patch will
> get almost a full minor release cycle to sit in the stable branches
> before release.

I did a read-through of these patches.

+ new_requests = entry->requests[forknum];
+ entry->requests[forknum] =
+ bms_join(new_requests, requests);

What happens if bms_join fails, too?

+        recover from the WAL after any failure is reported, preferrably

preferably.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company


Re: Postgres, fsync, and OSs (specifically linux)

From
Thomas Munro
Date:
On Fri, Nov 9, 2018 at 7:07 AM Robert Haas <robertmhaas@gmail.com> wrote:
> On Wed, Nov 7, 2018 at 9:41 PM Thomas Munro
> <thomas.munro@enterprisedb.com> wrote:
> > My plan is do a round of testing and review of this stuff next week
> > once the dust is settled on the current minor releases (including
> > fixing a few typos I just spotted and some word-smithing).  All going
> > well, I will then push the resulting patches to master and all
> > supported stable branches, unless other reviews or objections appear.
> > At some point not too far down the track I hope to be ready to
> > consider committing that other patch that will completely change all
> > of this code in the master branch, but in any case Craig's patch will
> > get almost a full minor release cycle to sit in the stable branches
> > before release.
>
> I did a read-through of these patches.
>
> + new_requests = entry->requests[forknum];
> + entry->requests[forknum] =
> + bms_join(new_requests, requests);
>
> What happens if bms_join fails, too?

My reasoning for choosing bms_join() is that it cannot fail, assuming
the heap is not corrupted.  It simply ORs the two bit-strings into
whichever is the longer input string, and frees the shorter input
string.  (In an earlier version I used bms_union(), this function's
non-destructive sibling, but then realised that it could fail to
allocate() causing us to lose track of a 1 bit).

Philosophical point: if pfree() throws, then bms_join() throws, but
(assuming AllocSetFree() implementation) it can only throw if the heap
is corrupted, eg elog(ERROR, "could not find block containing chunk
%p", chunk) and possibly other errors.  Of course it's impossible to
make guarantees of any kind in case of arbitrary corruption.  But
perhaps we could do this in a critical section, so errors are promoted
to PANIC.

> +        recover from the WAL after any failure is reported, preferrably
>
> preferably.

Thanks.

-- 
Thomas Munro
http://www.enterprisedb.com


Re: Postgres, fsync, and OSs (specifically linux)

From
Robert Haas
Date:
On Thu, Nov 8, 2018 at 3:04 PM Thomas Munro
<thomas.munro@enterprisedb.com> wrote:
> My reasoning for choosing bms_join() is that it cannot fail, assuming
> the heap is not corrupted.  It simply ORs the two bit-strings into
> whichever is the longer input string, and frees the shorter input
> string.  (In an earlier version I used bms_union(), this function's
> non-destructive sibling, but then realised that it could fail to
> allocate() causing us to lose track of a 1 bit).

Oh, OK.  I was assuming it was allocating.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company


Re: Postgres, fsync, and OSs (specifically linux)

From
Thomas Munro
Date:
On Fri, Nov 9, 2018 at 9:06 AM Robert Haas <robertmhaas@gmail.com> wrote:
> On Thu, Nov 8, 2018 at 3:04 PM Thomas Munro
> <thomas.munro@enterprisedb.com> wrote:
> > My reasoning for choosing bms_join() is that it cannot fail, assuming
> > the heap is not corrupted.  It simply ORs the two bit-strings into
> > whichever is the longer input string, and frees the shorter input
> > string.  (In an earlier version I used bms_union(), this function's
> > non-destructive sibling, but then realised that it could fail to
> > allocate() causing us to lose track of a 1 bit).
>
> Oh, OK.  I was assuming it was allocating.

I did some more testing using throw-away fault injection patch 0003.
I found one extra problem:  fsync_fname() needed data_sync_elevel()
treatment, because it is used in eg CheckPointCLOG().

With data_sync_retry = on, if you update a row, touch
/tmp/FileSync_EIO and try to checkpoint then the checkpoint fails, and
the cluster keeps running.  Future checkpoint attempts report the same
error about the same file, showing that patch 0001 works (we didn't
forget about the dirty file).  Then rm /tmp/FileSync_EIO, and the next
checkpoint should succeed.

With data_sync_retry = off (the default), the same test produces a
PANIC, showing that patch 0002 works.

It's similar if you touch /tmp/pg_sync_EIO instead.  That shows that
cases like fsync_fname("pg_xact") also cause PANIC when
data_sync_retry = off, but it hides the bug that 0001 fixes when
data_sync_retry = on, hence my desire to test the two different fault
injection points.

I think these patches are looking good now.  If I don't spot any other
problems or hear any objections, I will commit them tomorrow-ish.

-- 
Thomas Munro
http://www.enterprisedb.com

Attachment

Re: Postgres, fsync, and OSs (specifically linux)

From
Thomas Munro
Date:
On Fri, Nov 9, 2018 at 9:03 AM Thomas Munro
<thomas.munro@enterprisedb.com> wrote:
> On Fri, Nov 9, 2018 at 7:07 AM Robert Haas <robertmhaas@gmail.com> wrote:
> > On Wed, Nov 7, 2018 at 9:41 PM Thomas Munro
> > <thomas.munro@enterprisedb.com> wrote:
> > > My plan is do a round of testing and review of this stuff next week
> > > once the dust is settled on the current minor releases (including
> > > fixing a few typos I just spotted and some word-smithing).  All going
> > > well, I will then push the resulting patches to master and all
> > > supported stable branches, unless other reviews or objections appear.

...

On Sun, Nov 18, 2018 at 3:20 PM Thomas Munro
<thomas.munro@enterprisedb.com> wrote:
> I think these patches are looking good now.  If I don't spot any other
> problems or hear any objections, I will commit them tomorrow-ish.

Hearing no objections, pushed to all supported branches.

Thank you to Craig for all his work getting to the bottom of this, to
Andres for his open source diplomacy, and the Linux guys for their
change "errseq: Always report a writeback error once" which came out
of that.

Some more comments:

* The promotion of errors from close() to PANIC may or may not be
effective considering that it doesn't have interlocking with
concurrent checkpoints.  I'm not sure if it can really happen on local
file systems anyway... this may fall under the category of "making
PostgreSQL work reliably on NFS", a configuration that is not
recommended currently, and a separate project IMV.

* In 9.4 and 9.5 there is no checking of errors from
sync_file_range(), and I didn't add any for now.  It was claimed that
sync_file_range() without BEFORE/AFTER can't consume errors[1].
Errors are promoted in 9.6+ for consistency because we already looked
at the return code, so we won't long rely on that knowledge in the
long term.

* I personally believe it is safe to run with data_sync_retry = on on
any file system on FreeBSD, and ZFS on any operating system... but I
see no need to make recommendations about that in the documentation,
other than that you should investigate the behaviour of your operating
system if you really want to turn it on.

* A PANIC (and possibly ensuing crash restart loop if the I/O error is
not transient) is of course a very unpleasant failure mode, but it is
one that we already had for the WAL and control file.  So I'm not sure
I'd personally bother to run with the non-default setting even on a
system where I believe it to be safe (considering the low likelihood
that I/O failure is isolated to certain files); at best it probably
gives you a better experience if the fs underneath a non-default
tablespace dies.

* The GUC is provided primarily because this patch is so drastic in
its effect that it seems like we owe our users a way to disable it on
principle, and that seems to outweigh a desire not to add GUCs in
back-branches.

* If I/O errors happen, your system is probably toast and you need to
fail over or restore from backups, but at least we won't tell you any
lies about checkpoints succeeding.  In rare scenarios, perhaps
involving a transient failure of virtualised storage with thin
provisioning as originally described by Craig, the system may actually
be able to continue running, and with this change we should now be
able to avoid data loss by recovering from the WAL.

* As noted the commit message, this isn't quite the end of the story.
See the fsync queue redesign thread[2], WIP for master only.

[1] https://www.postgresql.org/message-id/20180430160945.5s5qfoqryhtmugxl%40alap3.anarazel.de
[2] https://www.postgresql.org/message-id/flat/CAEepm=2gTANm=e3ARnJT=n0h8hf88wqmaZxk0JYkxw+b21fNrw@mail.gmail.com

-- 
Thomas Munro
http://www.enterprisedb.com