Thread: Why does replication need the old history file?
Hackers, Sequence of events: 1. PITR backup of server on timeline 2. 2. Restored the backup to a new server, new-master. 3. Restored the backup to another new server, new-replica. 4. Started and promoted new-master (now on Timeline 3). 5. Started new-replica, connecting over streaming to new-master. 6. Get error message: 2015-06-11 12:24:14.503 PDT,,,7465,,5579e05e.1d29,1,,2015-06-11 12:24:14 PDT,,0,LOG,00000,"fetching timeline history file for timeline 2 from primary server",,,,,,,,,"" 2015-06-11 12:24:14.503 PDT,,,7465,,5579e05e.1d29,2,,2015-06-11 12:24:14 PDT,,0,FATAL,XX000,"could not receive timeline history file from the primary server: ERROR: could not open file ""pg_xlog/00000002.history"": No such file or directory Questions: A. Why does the replica need 00000002.history? Shouldn't it only need 00000003.history? B. Did something change in this regard in 9.3.6, 9.3.7 or 9.3.8? It was working in our previous setup, on 9.3.5, although that could have just been that the history file hadn't been removed from the backups yet. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com
On Fri, Jun 12, 2015 at 4:56 AM, Josh Berkus <josh@agliodbs.com> wrote: > Hackers, > > Sequence of events: > > 1. PITR backup of server on timeline 2. > > 2. Restored the backup to a new server, new-master. > > 3. Restored the backup to another new server, new-replica. > > 4. Started and promoted new-master (now on Timeline 3). > > 5. Started new-replica, connecting over streaming to new-master. > > 6. Get error message: > > 2015-06-11 12:24:14.503 PDT,,,7465,,5579e05e.1d29,1,,2015-06-11 12:24:14 > PDT,,0,LOG,00000,"fetching timeline history file for timeline 2 from > primary server",,,,,,,,,"" > 2015-06-11 12:24:14.503 PDT,,,7465,,5579e05e.1d29,2,,2015-06-11 12:24:14 > PDT,,0,FATAL,XX000,"could not receive timeline history file from the > primary server: ERROR: could not open file > ""pg_xlog/00000002.history"": No such file or directory > > Questions: > > A. Why does the replica need 00000002.history? Shouldn't it only need > 00000003.history? From where is the base backup taken in case of the node started at 5? > B. Did something change in this regard in 9.3.6, 9.3.7 or 9.3.8? It was > working in our previous setup, on 9.3.5, although that could have just > been that the history file hadn't been removed from the backups yet. At quick glance, I can see nothing in xlog.c between those releases. -- Michael
On Fri, Jun 12, 2015 at 5:18 PM, Michael Paquier
<michael.paquier@gmail.com> wrote:
> On Fri, Jun 12, 2015 at 4:56 AM, Josh Berkus <josh@agliodbs.com> wrote:
>> Hackers,
>>
>> Sequence of events:
>>
>> 1. PITR backup of server on timeline 2.
>>
>> 2. Restored the backup to a new server, new-master.
>>
>> 3. Restored the backup to another new server, new-replica.
>>
>> 4. Started and promoted new-master (now on Timeline 3).
>>
>> 5. Started new-replica, connecting over streaming to new-master.
>>
>> 6. Get error message:
>>
>> 2015-06-11 12:24:14.503 PDT,,,7465,,5579e05e.1d29,1,,2015-06-11 12:24:14
>> PDT,,0,LOG,00000,"fetching timeline history file for timeline 2 from
>> primary server",,,,,,,,,""
>> 2015-06-11 12:24:14.503 PDT,,,7465,,5579e05e.1d29,2,,2015-06-11 12:24:14
>> PDT,,0,FATAL,XX000,"could not receive timeline history file from the
>> primary server: ERROR: could not open file
>> ""pg_xlog/00000002.history"": No such file or directory
>>
>> Questions:
>>
>> A. Why does the replica need 00000002.history? Shouldn't it only need
>> 00000003.history?
>
> From where is the base backup taken in case of the node started at 5?
The related source code comment says
/* * Get any missing history files. We do this always, even when we're * not interested in that
timeline,so that if we're promoted to * become the master later on, we don't select the same timeline that
* was already used in the current master. This isn't bullet-proof - * you'll need some external software to
manageyour cluster if you * need to ensure that a unique timeline id is chosen in every case, * but let's
avoidthe confusion of timeline id collisions where we * can. */
WalRcvFetchTimeLineHistoryFiles(startpointTLI,primaryTLI);
>
>> B. Did something change in this regard in 9.3.6, 9.3.7 or 9.3.8? It was
>> working in our previous setup, on 9.3.5, although that could have just
>> been that the history file hadn't been removed from the backups yet.
>
> At quick glance, I can see nothing in xlog.c between those releases.
Yep, I could reproduce the "trouble" even in 9.3.5 in my laptop.
Regards,
--
Fujii Masao
>>> Questions: >>> >>> A. Why does the replica need 00000002.history? Shouldn't it only need >>> 00000003.history? >> >> From where is the base backup taken in case of the node started at 5? It is the same backup used to restore the master, restored to a point in time 5 minutes earlier just to make sure the replica isn't ahead of the master. > > The related source code comment says > > /* > * Get any missing history files. We do this always, even when we're > * not interested in that timeline, so that if we're promoted to > * become the master later on, we don't select the same timeline that > * was already used in the current master. This isn't bullet-proof - > * you'll need some external software to manage your cluster if you > * need to ensure that a unique timeline id is chosen in every case, > * but let's avoid the confusion of timeline id collisions where we > * can. > */ > WalRcvFetchTimeLineHistoryFiles(startpointTLI, primaryTLI); So this seems to be something we're doing "just in case" which is preventing a useful way to spin up large master/replica clusters from PITR backup. Might this be something we want to change, and simply error that we can't find the history file instead of FATAL? -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com
>> The related source code comment says >> >> /* >> * Get any missing history files. We do this always, even when we're >> * not interested in that timeline, so that if we're promoted to >> * become the master later on, we don't select the same timeline that >> * was already used in the current master. This isn't bullet-proof - >> * you'll need some external software to manage your cluster if you >> * need to ensure that a unique timeline id is chosen in every case, >> * but let's avoid the confusion of timeline id collisions where we >> * can. >> */ >> WalRcvFetchTimeLineHistoryFiles(startpointTLI, primaryTLI); > > So this seems to be something we're doing "just in case" which is > preventing a useful way to spin up large master/replica clusters from > PITR backup. Might this be something we want to change, and simply > warn that we can't find the history file instead of ERROR? Having experimented with this further, it seems to me that either we have a functionality bug here, or a documentation bug. Currently our docs do not treat the history files as critical, so much so that authors of backup tools, like Barman, treat them as expireable. But replication apparently regards them as being as essiential as CLOG files. The result is that, in some cases, a Barman restored database can never participate in replication again. So either we should remove the requirement to have old history files to start replication, *or* we should document somewhere to never, ever delete history files and to retain them permanently for backups. I know what I'd prefer, but I'd like to hear the case for what could go wrong if we don't retain the old history file requirement. BTW, if retaining history files forever is a requirement, then maybe they should go in their own directory in the future. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com