Re: [patch] pg_copy - a command for reliable WAL archiving - Mailing list pgsql-hackers

From Alvaro Herrera
Subject Re: [patch] pg_copy - a command for reliable WAL archiving
Date
Msg-id 20140819214205.GG6817@eldon.alvh.no-ip.org
Whole thread Raw
In response to Re: [patch] pg_copy - a command for reliable WAL archiving  ("MauMau" <maumau307@gmail.com>)
Responses Re: [patch] pg_copy - a command for reliable WAL archiving
Re: [patch] pg_copy - a command for reliable WAL archiving
List pgsql-hackers
MauMau wrote:

> With that said, copying to a temporary file like <dest>.tmp and
> renaming it to <dest> sounds worthwhile even as a basic copy
> utility.  I want to avoid copying to a temporary file with a fixed
> name like _copy.tmp, because some advanced utility may want to run
> multiple instances of pg_copy to copy several files into the same
> directory simultaneously.  However, I'm afraid multiple <dest>.tmp
> files might continue to occupy disk space after canceling copy or
> power failure in some use cases, where the copy of the same file
> won't be retried.  That's also the reason why I chose to not use a
> temporary file like cp/copy.

Is there a way to create a link to a file which only exists as an open
file descriptor?   If there was, you could create a temp file, open an
fd, then delete the file.  That would remove the issue with files being
leaked due to failures of various kinds.

Also, it's been mentioned that this utility might be useful for
restore_command.  That sounds good I guess, but need to keep the
RECOVERYXLOG trick in mind.  I remember a case of stalled replay because
the restore command was writing to RECOVERYXLOG.gz and ungzipping, and
the unlink("RECOVERYXLOG") call failed after a partial copy and so did
the copy from the archive.  (Removing the borked RECOVERYXLOG.gz fixed
it.)

-- 
Álvaro Herrera                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services



pgsql-hackers by date:

Previous
From: John Lumby
Date:
Subject: Re: Extended Prefetching using Asynchronous IO - proposal and patch
Next
From: Heikki Linnakangas
Date:
Subject: Re: PQgetssl() and alternative SSL implementations