Re: BUG: *FF WALs under 9.2 (WAS: .ready files appearing on slaves) - Mailing list pgsql-hackers

On 10/23/2014 08:59 AM, Fujii Masao wrote:
> On Mon, Oct 20, 2014 at 3:26 PM, Michael Paquier
> <michael.paquier@gmail.com> wrote:
>> On Fri, Oct 17, 2014 at 10:37 PM, Michael Paquier
>> <michael.paquier@gmail.com> wrote:
>>>
>>> On Fri, Oct 17, 2014 at 9:23 PM, Fujii Masao <masao.fujii@gmail.com> wrote:
>>>>
>>>> In this case, the patch seems to make the restartpoint recycle even WAL files
>>>> which have .ready files and will have to be archived later. Thought?
>>>
>>> The real problem currently is that it is possible to have a segment file not marked as .done during recovery when
streamconnection is abruptly cut when this segment is switched, marking it as .ready in archive_status and simply
lettingthis segment in pg_xlog because it will neither be recycled nor removed. I have not been able to look much at
thiscode these days, so I am not sure how invasive it would be in back-branches, but perhaps we should try to improve
codesuch as when a segment file is switched and connection to the is cut, we guarantee that this file is completed and
markedas .done.
 
>>
>> I have spent more time on that, with a bit more of underground work...
>> First, the problem can be reproduced most of the time by running this
>> simple command:
>> psql -c 'select pg_switch_xlog()'; pg_ctl restart -m immediate
>
> What about fixing this problem directly? That is, we can make walreceiver
> check whether the end of last received WAL data is the end of current WAL file
> or not, and then close the WAL file and create .done file if the test is true.
>
> This is not a perfect solution. If the standby crashes during very
> short interval
> (i.e., after closing the WAL file and before creating .done file), the problem
> would happen again. But it can really rarely happen, so I don't think that it's
> worth fixing the corner case at least in back-branches. Of course, we can
> find out the "perfect" solution for the master, though.

Sounds reasonable, for back-branches. Although I'm still worried we 
might miss some corner-case unless we go with a more wholesale solution.

At least for master, we should consider changing the way the archiving 
works so that we only archive WAL that was generated in the same server. 
I.e. we should never try to archive WAL files belonging to another timeline.

I just remembered that we discussed a different problem related to this 
some time ago, at 
http://www.postgresql.org/message-id/20131212.110002.204892575.horiguchi.kyotaro@lab.ntt.co.jp. 
The conclusion of that was that at promotion, we should not archive the 
last, partial, segment from the old timeline. Also, we could make the 
archive recovery more precise in what WAL files it tries to restore. For 
example, if we're doing recovery with target timeline 4, but the 
timeline history says that segment AB came from timeline 3 (i.e. the 
switch to timeline 4 happened within a later segment), we should not try 
to restore segment AB from timeline 4. It shouldn't exist, so trying to 
restore it is a waste of cycles, but more importantly, if there's some 
sort of confusion with the timelines, and the file exists anyway, we 
should not restore it. Also, if the timeline history says e.g. that we 
switched from TLI 3 to 4 at WAL position 0/AD123456, and we restore 
segment AD from timeline 3, we should stop recovering that segment at 
position 0/AD123456 and try to find the AD segment from timeline 4. We 
currently only do that when streaming, but we should also do that when 
restoring from archive.

In summary, let's do something small for back-branches, like what you 
suggested. But for master, let's do bigger changes to the timeline 
handling. Would you like to write a patch for the back-branches, if I 
start working on master?

- Heikki