Re: Reliable WAL file shipping over unreliable network - Mailing list pgsql-admin

From Rui DeSousa
Subject Re: Reliable WAL file shipping over unreliable network
Date
Msg-id CF30D15A-BD42-42A7-9AD3-BB69AF9E8047@icloud.com
Whole thread Raw
In response to Re: Reliable WAL file shipping over unreliable network  (scott ribe <scott_ribe@elevated-dev.com>)
Responses Re: Reliable WAL file shipping over unreliable network  (Nagy László Zsolt <gandalf@shopzeus.com>)
Re: Reliable WAL file shipping over unreliable network  (scott ribe <scott_ribe@elevated-dev.com>)
List pgsql-admin

Just use "-ac”;  you want -c option to ensure no data corruption during the transfer.  Do not delete the file; let Postgres manage that.

Here is a snippet from I script I use for archiving.  You also want to make your script returns failure or success correctly.  

# SSH Command and options
SSH_CMD="ssh -o ServerAliveInterval=20 $ARCH_SERVER"
STS=3

OUTPUT=$(rsync -ac --rsync-path="mkdir -p $ARCH_DIR && rsync" $XLOGFILE $ARCH_SERVER:$ARCH_DIR/$WALFILE)
if [ $? == 0 ]; then 
   STS=0
fi

exit $STS

On Feb 28, 2018, at 11:38 AM, scott ribe <scott_ribe@elevated-dev.com> wrote:

On Feb 28, 2018, at 8:46 AM, Nagy László Zsolt <gandalf@shopzeus.com> wrote:

Another problem with rsync might be that it does not know if a WAL file
on the source side is growing (e.g. being written) and it might start
copy that before it is fully flushed to disk. I see this as a big
problem, but I don't have experience.

You use rsync in the archive command, not by itself. Postgres does not try to copy the file until it is ready.

E.g. move the file into the archive dir only after it has been fully
copied to the destination volume. But then we still have the problem of
partially written files on the slave side.

No, you do not have that problem at all.

Given an unreliable network, you probably do want to use --partial


--
Scott Ribe
https://www.linkedin.com/in/scottribe/
(303) 722-0567



pgsql-admin by date:

Previous
From: Jimmy Augustine
Date:
Subject: Re: PostgreSQL 9.6 Temporary files
Next
From: Nagy László Zsolt
Date:
Subject: Re: Reliable WAL file shipping over unreliable network