Thread: Does PostgreSQL use atomic file creation of FS?
Hello. Does PostgreSQL use atomic file creation on FS? How does PostgreSQL catch situation when system crashes between open call and write call? I am interesting in this because I would like use PostgreSQL on network file system. Thank you.
On Wed, Dec 12, 2018 at 02:48:12PM +0300, Dmitry Lazurkin wrote:
> Does PostgreSQL use atomic file creation on FS? How does PostgreSQL
> catch situation when system crashes between open call and write call? I
> am interesting in this because I would like use PostgreSQL on network
> file system.
I doubt we can get more certainty than this:
https://www.postgresql.org/docs/devel/creating-cluster.html#CREATING-CLUSTER-NFS
Best,
Karsten
--
GPG 40BE 5B0E C98E 1713 AFA6 5BC0 3BEA AC80 7D4F C89B
Thank you. But I have read this. I said about network file system only for example. I would like to known how PostgreSQL handles this specific case (of course if someone knowns a answer): fd = open(file, "w"); write(fd, data); // crash and now I have empty file which isn't correct fsync(fd); PS. I think PostgreSQL doesn't have this problem. On 12/12/18 15:37, Karsten Hilbert wrote: > On Wed, Dec 12, 2018 at 02:48:12PM +0300, Dmitry Lazurkin wrote: > >> Does PostgreSQL use atomic file creation on FS? How does PostgreSQL >> catch situation when system crashes between open call and write call? I >> am interesting in this because I would like use PostgreSQL on network >> file system. > I doubt we can get more certainty than this: > > https://www.postgresql.org/docs/devel/creating-cluster.html#CREATING-CLUSTER-NFS > > Best, > Karsten
Dmitry Lazurkin wrote: > Does PostgreSQL use atomic file creation on FS? How does PostgreSQL > catch situation when system crashes between open call and write call? I > am interesting in this because I would like use PostgreSQL on network > file system. If there is a crash, the file would be left behind. This is slightly annoying, but shouldn't be a major problem. Persisting such information in a crash-safe way would require fsyncs that hurt I/O performance. Yours, Laurenz Albe -- Cybertec | https://www.cybertec-postgresql.com
On Wed, Dec 12, 2018 at 11:52 PM Dmitry Lazurkin <dilaz03@gmail.com> wrote: > Thank you. But I have read this. I said about network file system only > for example. I would like to known how PostgreSQL handles this specific > case (of course if someone knowns a answer): > > fd = open(file, "w"); > write(fd, data); > // crash and now I have empty file which isn't correct > fsync(fd); > > PS. I think PostgreSQL doesn't have this problem. It depends on the context, but in general PostgreSQL knows about that sort of thing. When the cluster shuts down, it records that it shut down cleanly, meaning that everything that should be on disk is on disk. When you start the cluster up, if it sees that it didn't shut down cleanly, it enters recovery. During recovery it tolerates files being too short while it's replaying the WAL to get back to a consistent state. See the comment in mdread() for example: https://github.com/postgres/postgres/blob/master/src/backend/storage/smgr/md.c#L755 It's called "write-ahead log" because we log our intention before we write to data files (and make sure it's on disk first), so we'll always replay the same effects again if we're interrupted. The WAL is a magic source of reliability (we can do it again if things go wrong) and also performance (IO becomes serial, optimised for the storage hardware). https://www.postgresql.org/docs/current/wal-intro.html -- Thomas Munro http://www.enterprisedb.com