Re: [patch] pg_copy - a command for reliable WAL archiving - Mailing list pgsql-hackers

From Alvaro Herrera
Subject Re: [patch] pg_copy - a command for reliable WAL archiving
Date
Msg-id 20140820135056.GB7648@eldon.alvh.no-ip.org
Whole thread Raw
In response to Re: [patch] pg_copy - a command for reliable WAL archiving  (Andres Freund <andres@2ndquadrant.com>)
Responses Re: [patch] pg_copy - a command for reliable WAL archiving
List pgsql-hackers
Andres Freund wrote:
> On 2014-08-19 17:42:06 -0400, Alvaro Herrera wrote:
> > MauMau wrote:
> > 
> > > With that said, copying to a temporary file like <dest>.tmp and
> > > renaming it to <dest> sounds worthwhile even as a basic copy
> > > utility.  I want to avoid copying to a temporary file with a fixed
> > > name like _copy.tmp, because some advanced utility may want to run
> > > multiple instances of pg_copy to copy several files into the same
> > > directory simultaneously.  However, I'm afraid multiple <dest>.tmp
> > > files might continue to occupy disk space after canceling copy or
> > > power failure in some use cases, where the copy of the same file
> > > won't be retried.  That's also the reason why I chose to not use a
> > > temporary file like cp/copy.
> > 
> > Is there a way to create a link to a file which only exists as an open
> > file descriptor?   If there was, you could create a temp file, open an
> > fd, then delete the file.  That would remove the issue with files being
> > leaked due to failures of various kinds.
> 
> Isn't this a solution looking for a problem? We're using tempfiles in
> dozens of other places and I really don't see why this is the place to
> stop doing so. Just copy to <dest>.tmp and move it into place. If things
> crash during that, the command will be repeated shortly afterwards again
> *anyway*. Let's not get into platform specific games here.

The issue is what happens if there's a crash while the temp file is in
the middle of being filled.  I agree there are bigger problems to solve
in this area though.  Yes, I agree that a fixed name such as _copy.tmp
is a really bad choice for several reasons.

-- 
Álvaro Herrera                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services



pgsql-hackers by date:

Previous
From: Andres Freund
Date:
Subject: Re: [patch] pg_copy - a command for reliable WAL archiving
Next
From: Greg Stark
Date:
Subject: Re: [patch] pg_copy - a command for reliable WAL archiving