On Tue, 4 Oct 2005, Simon Riggs wrote:
>> Now here's the problem...the servers
>> switched roles on Sep 21. We switched them back a little while after that.
>> When I went to replay the WAL files using the Sep 15 base backup, it happily
>> played back the WAL files to Sep 21 and stopped. I presume that this means
>> whenever the servers switch roles I need to create a fresh base backup?
>
> Yes, but more generally if I lost one node I would always be inclined to
> fully backup the remaining one just in case.
I believe they were switched not because one went down, but to add more ram to
the primary. They switched back just peachy at that time.
>> Is there any possible way to replay the other WAL files after the Sep 21
>> switchover/switchback? I'm going to guess this is similar to having another
>> timeline.
>
> That may be your exact case, it depends upon how you did recovery. Both
> systems think they are the same one, so you need to be careful.
>
> Look for the last log file of the went-down node. That file should exist
> twice, once as finally written by the went-down node and once as
> continued to be written to by the stayed-up node after switchover. It
> sounds like the wrong file was used to recover with and as a result
> stopped recovery on Sep 21. But if you're running with the system now,
> you'll need to do an intermediate rebuild and recover the data manually.
What would constitute an intermediate rebuild? Of course the system is up and
live and having data added to it. How would one restore from multiple
timelines?
> (I recommend testing recovery procedures before you go live and then at
> least every 6 months, to ensure that they actually still work when you
> need 'em. Trying to do a recovery with any confidence is not easy at 4am
> on a Sunday morning under maximum stress, from experience.)
>
I absolutely agree..unfortunately, it's only been 3 months since the last
test. :-(
--
Jeff Frost, Owner <jeff@frostconsultingllc.com>
Frost Consulting, LLC http://www.frostconsultingllc.com/
Phone: 650-780-7908 FAX: 650-649-1954
