Thread: pgsql: Don't archive bogus recycled or preallocated files after timelin

Don't archive bogus recycled or preallocated files after timeline switch.

After a timeline switch, we would leave behind recycled WAL segments that
are in the future, but on the old timeline. After promotion, and after they
become old enough to be recycled again, we would notice that they don't have
a .ready or .done file, create a .ready file for them, and archive them.
That's bogus, because the files contain garbage, recycled from an older
timeline (or prealloced as zeros). We shouldn't archive such files.

This could happen when we're following a timeline switch during replay, or
when we switch to new timeline at end-of-recovery.

To fix, whenever we switch to a new timeline, scan the data directory for
WAL segments on the old timeline, but with a higher segment number, and
remove them. Those don't belong to our timeline history, and are most
likely bogus recycled or preallocated files. They could also be valid files
that we streamed from the primary ahead of time, but in any case, they're
not needed to recover to the new timeline.

Branch
------
master

Details
-------
http://git.postgresql.org/pg/commitdiff/b2a5545bd63fc94a71b1e97ecdd03c605d97a438

Modified Files
--------------
src/backend/access/transam/xlog.c        |  280 ++++++++++++++++++++----------
src/backend/access/transam/xlogarchive.c |   19 ++
src/include/access/xlog_internal.h       |    1 +
3 files changed, 211 insertions(+), 89 deletions(-)

Heikki Linnakangas <heikki.linnakangas@iki.fi> writes:
> To fix, whenever we switch to a new timeline, scan the data directory for
> WAL segments on the old timeline, but with a higher segment number, and
> remove them.

Wait a minute.  Didn't you just break PITR scenarios?  The *entire point*
of the multiple-timeline mechanism is that you can start a new WAL history
from a point in the past without destroying WAL history after that point
on the old timeline.

I've not read the patch, so maybe the code is all right, but this
description most certainly is not.

            regards, tom lane

On 04/14/2015 02:24 AM, Tom Lane wrote:
> Heikki Linnakangas <heikki.linnakangas@iki.fi> writes:
>> To fix, whenever we switch to a new timeline, scan the data directory for
>> WAL segments on the old timeline, but with a higher segment number, and
>> remove them.
>
> Wait a minute.  Didn't you just break PITR scenarios?  The *entire point*
> of the multiple-timeline mechanism is that you can start a new WAL history
> from a point in the past without destroying WAL history after that point
> on the old timeline.

It deletes the files belonging to other timelines from pg_xlog only. The
WAL archive is not touched.

- Heikki

Heikki Linnakangas <hlinnaka@iki.fi> writes:
> On 04/14/2015 02:24 AM, Tom Lane wrote:
>> Wait a minute.  Didn't you just break PITR scenarios?  The *entire point*
>> of the multiple-timeline mechanism is that you can start a new WAL history
>> from a point in the past without destroying WAL history after that point
>> on the old timeline.

> It deletes the files belonging to other timelines from pg_xlog only. The
> WAL archive is not touched.

How do we know those files are in the WAL archive?

            regards, tom lane

On 04/14/2015 05:32 PM, Tom Lane wrote:
> Heikki Linnakangas <hlinnaka@iki.fi> writes:
>> On 04/14/2015 02:24 AM, Tom Lane wrote:
>>> Wait a minute.  Didn't you just break PITR scenarios?  The *entire point*
>>> of the multiple-timeline mechanism is that you can start a new WAL history
>>> from a point in the past without destroying WAL history after that point
>>> on the old timeline.
>
>> It deletes the files belonging to other timelines from pg_xlog only. The
>> WAL archive is not touched.
>
> How do we know those files are in the WAL archive?

Presumably that's where they came from if you're doing PITR. Or they
were copied into pg_xlog manually from somewhere else.

After PITR, if you decide that you didn't like the result and perform
another recovery, you have to restore from a base backup again, which
will wipe out pg_xlog anyway.

Also, if the recovery continued longer, those files would be deleted
from pg_xlog anyway, at a later restartpoint.

- Heikki