Re: [RFC] What should we do for reliable WAL archiving? - Mailing list pgsql-hackers

From: "Mitsumasa KONDO" <kondo.mitsumasa@gmail.com>
> 2014-03-17 21:12 GMT+09:00 Fujii Masao <masao.fujii@gmail.com>:
>
>> On Mon, Mar 17, 2014 at 10:20 AM, Robert Haas <robertmhaas@gmail.com>
>> wrote:
>> > On Sun, Mar 16, 2014 at 6:23 AM, MauMau <maumau307@gmail.com> wrote:
>> >> * Improve the example in the documentation.
>> >> But what command can we use to reliably sync just one file?
>> >>
>> >> * Provide some command, say pg_copy, which copies a file synchronously
>> by
>> >> using fsync(), and describes in the doc something like "for simple use
>> >> cases, you can use pg_copy as the standard reliable copy command."
>> >
>> > +1.  This won't obviate the need for tools to manage replication, but
>> > it would make it possible to get the simplest case right without
>> > guessing.
>>
>> +1, too.
>>
>> And, what about making pg_copy call posix_fadvise(DONT_NEED) against the
>> archived file after the copy? Also It might be good idea to support the
>> direct
>> copy of the file to avoid wasting the file cache.
>
> Use direct_cp.
> http://directcp.sourceforge.net/direct_cp.html

Thank you all for giving favorable responses and interesting ideas.
Then, I think I'll do:

* Create pg_copy in C so that it can be used on Windows as well as on 
UNIX/Linux.  It just copies one file.  Its source code is located in 
src/bin/pg_copy/.  Please recommend a better name if you have one in mind.

* Add a reference page for pg_copy in the chapter "Server applications". 
Modify the section for continuous archiving to recommend pg_copy for simple 
use cases as the standard command.

* pg_copy calls posix_fadvise(DONT_NEED) on the destination file.

* pg_copy passes O_DIRECT flag when opening the destination file 
when --directio or -d option is specified.  O_DIRECT is not used by default 
because it may not be available on some file systems, as well as it might 
cause trouble on older platforms such as RHEL4/5.  pg_copy does not use 
O_DIRECT for the source file so that it can copy the data from the 
filesystem cache, which is just written by postgres.

Could you give me your opinions before starting the work, including the 
following?

* Should I refactor the functions (copy_file, copydir, etc.) in 
src/backend/storage/file/copydir.c so that they can also be used for 
frontends?  If so, which of src/port or src/common/ is the right place to 
put copydir.c in?

* Should I complete the work before 9.4 beta so that it will be available 
starting with 9.4?  I think so because it is a basic capability to archive 
transaction logs safely (although the time may not allow me to do this).

Regards
MauMau