Re: BUG #11603: replication, pg_basebackup and high load - Mailing list pgsql-bugs

From Heikki Linnakangas
Subject Re: BUG #11603: replication, pg_basebackup and high load
Date
Msg-id 5436BB3B.80500@vmware.com
Whole thread Raw
In response to BUG #11603: replication, pg_basebackup and high load  (mdglange@gmail.com)
List pgsql-bugs
On 10/08/2014 04:19 PM, mdglange@gmail.com wrote:
> The following bug has been logged on the website:
>
> Bug reference:      11603
> Logged by:          Michiel
> Email address:      mdglange@gmail.com
> PostgreSQL version: 9.4beta2
> Operating system:   Linux
> Description:
>
> The test I did involved the following: a master database with two slaves. On
> the master two replication slots have been configured as per the
> documentation. One slave active before I put some "heavy" load (the
> environment is scaled such, that inserting a few gigabytes of insert
> statements is a heavy load. This is on purpose)
>
> During this load I started a pg_basebackup from the master, immediately
> followed (pg_basebackup && mv recovery.conf $PG_DATA/ && pg_ctl start) by
> placing the prepared recovery.conf and starting this slave. The
> pg_basebackup took a few hours (as expected) but starting this latter slave
> would not work because the WALs were no longer available.
>
> I'd expect to see that pg_basebackup restores up to the last WAL, so that
> regardless of the load and changes done on the (master) database replication
> picks up.

After pg_basebackup, the system needs to have all the WAL available from
the time the backup *started*, unfortunately. Try using pg_basebackup's
"--xlog-method=stream" option. That way it streams the WAL at the same
time the backup is taken, making it less likely that the master will
recycle the segments too quickly. Even that is not bullet-proof, though;
if there's a network hickup or something that makes the backup process
to stall for long enough, the master might still recycle the segments
that the backup would need.

Or, set up WAL archiving, and use a restore_command in the recovery.conf
file to pull the files from the archive.

I didn't understand what replication slots have to do with this though;
sorry if I misunderstood the whole thing..

- Heikki

pgsql-bugs by date:

Previous
From: Heikki Linnakangas
Date:
Subject: Re: Client deadlocks when connecting via ssl
Next
From: Andres Freund
Date:
Subject: Re: BUG #11603: replication, pg_basebackup and high load