Thread: Allow users to choose what happens when recovery target is not reached

Hi,

Currently, the server shuts down with a FATAL error (added by commit
[1]) when the recovery target isn't reached. This can cause a server
availability problem, especially in case of disaster recovery (geo
restores) where the primary was down and the user is doing a PITR on a
server lying in another region where it had missed to receive few of
the last WAL files required to reach the recovery target. In this
case, users might want the server to be available rather than a no
server. With the commit [1], there's no way to achieve what users
wanted.

There can be many reasons for the last few WAL files not reaching the
target server where the user is performing the PITR. The primary may
have been down before archiving the last few WAL files to the archive
locations, or archive command fails for whatever reasons or network
latency from primary to archive location and archive location to the
target server, or recovery command on the target server fails or users
may have chosen some wrong/futuristic recovery targets etc. If the
PITR fails with FATAL error and we may ask them to restart the server,
but imagine the wastage of compute resources - if there are a 1 TB of
WAL files to be replayed and just last 16MB WAL file is missing,
everything has to be replayed from the beginning.

Here's a proposal(and a patch) to have a GUC so that users can choose
either to emit a warning and promote or shutdown with FATAL error (as
default) when recovery target isn't reached. In reality, users can
choose to shutdown with FATAL error, if strict consistency is the
necessity, otherwise they can choose to get promoted, if availability
is preferred. There is some discussion around this idea in [2].

Thoughts?

[1] - commit dc788668bb269b10a108e87d14fefd1b9301b793
Author: Peter Eisentraut <peter@eisentraut.org>
Date:   Wed Jan 29 15:43:32 2020 +0100

    Fail if recovery target is not reached

    Before, if a recovery target is configured, but the archive ended
    before the target was reached, recovery would end and the server would
    promote without further notice.  That was deemed to be pretty wrong.
    With this change, if the recovery target is not reached, it is a fatal
    error.

    Based-on-patch-by: Leif Gunnar Erlandsen <leif@lako.no>
    Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com>
    Discussion:
https://www.postgresql.org/message-id/flat/993736dd3f1713ec1f63fc3b653839f5@lako.no

[2] - https://www.postgresql.org/message-id/b334d61396e6b0657a63dc38e16d429703fe9b96.camel%40j-davis.com

Regards,
Bharath Rupireddy.

On Fri, Nov 12, 2021 at 6:14 PM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
>
> Currently, the server shuts down with a FATAL error (added by commit
> [1]) when the recovery target isn't reached. This can cause a server
> availability problem, especially in case of disaster recovery (geo
> restores) where the primary was down and the user is doing a PITR on a
> server lying in another region where it had missed to receive few of
> the last WAL files required to reach the recovery target. In this
> case, users might want the server to be available rather than a no
> server. With the commit [1], there's no way to achieve what users
> wanted.

if users don't mind if the recovery target is reached or not isn't it
better to simply don't specify a target and let the recovery go as far
as possible?

On Fri, Nov 12, 2021 at 4:09 PM Julien Rouhaud <rjuju123@gmail.com> wrote:
>
> On Fri, Nov 12, 2021 at 6:14 PM Bharath Rupireddy
> <bharath.rupireddyforpostgres@gmail.com> wrote:
> >
> > Currently, the server shuts down with a FATAL error (added by commit
> > [1]) when the recovery target isn't reached. This can cause a server
> > availability problem, especially in case of disaster recovery (geo
> > restores) where the primary was down and the user is doing a PITR on a
> > server lying in another region where it had missed to receive few of
> > the last WAL files required to reach the recovery target. In this
> > case, users might want the server to be available rather than a no
> > server. With the commit [1], there's no way to achieve what users
> > wanted.
>
> if users don't mind if the recovery target is reached or not isn't it
> better to simply don't specify a target and let the recovery go as far
> as possible?

Users will always be optimistic and set a recovery target and try to
reach it, but somehow the few of the WAL files haven't arrived (for
whatever the reasons) the PITR target server, imagine if their primary
isn't available too, then with the proposal I made, they can choose to
have at least an available target server rather than a FATALly failed
one.

Regards,
Bharath Rupireddy.

On Sat, Nov 13, 2021 at 11:00 AM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
>
> Users will always be optimistic and set a recovery target and try to
> reach it, but somehow the few of the WAL files haven't arrived (for
> whatever the reasons) the PITR target server, imagine if their primary
> isn't available too, then with the proposal I made, they can choose to
> have at least an available target server rather than a FATALly failed
> one.

If your primary server isn't available, why would you want a recovery
target in the first place?  I just don't understand in which case
someone would want to setup a recovery target and wouldn't care if the
recovery wasn't reached, especially if it can be off by GB / days of
data.

It seems like it could have the opposite effect of what you want most
of the time.  What if for some reason the restore_command is flawed,
and you end up starting your server because it couldn't restore WAL
that are actually available?  You would have to restart from scratch
and waste more time than if you didn't use this.

It look like what you actually want is some kind of a target window,
but the window you currently propose is a hardcoded (consistency,
given target], and it seems too dangerous to be useful.

On Sat, Nov 13, 2021 at 9:45 AM Julien Rouhaud <rjuju123@gmail.com> wrote:
>
> On Sat, Nov 13, 2021 at 11:00 AM Bharath Rupireddy
> <bharath.rupireddyforpostgres@gmail.com> wrote:
> >
> > Users will always be optimistic and set a recovery target and try to
> > reach it, but somehow the few of the WAL files haven't arrived (for
> > whatever the reasons) the PITR target server, imagine if their primary
> > isn't available too, then with the proposal I made, they can choose to
> > have at least an available target server rather than a FATALly failed
> > one.
>
> If your primary server isn't available, why would you want a recovery
> target in the first place?  I just don't understand in which case
> someone would want to setup a recovery target and wouldn't care if the
> recovery wasn't reached, especially if it can be off by GB / days of
> data.
>
> It seems like it could have the opposite effect of what you want most
> of the time.  What if for some reason the restore_command is flawed,
> and you end up starting your server because it couldn't restore WAL
> that are actually available?  You would have to restart from scratch
> and waste more time than if you didn't use this.

Firstly, the proposed patch adds no new behaviour as such, it just
gives the ability that is existing today on v12 and below (prior to
commit dc78866 which went into v13 and later).

I think performing PITR is the user's wish - whether the primary is
available or not, it is completely the user's choice. The user might
start the PITR, when the primary is available, thinking that it sends
all the WAL files required for achieving recovery target. But imagine
a disaster happens and the primary server crashes, say the recovery
has replayed a huge bunch of WAL records (a TB may be), and the
primary failed without sending the last one or few WAL files, should
the PITR target server be failing this case after replaying a huge
bunch of WAL records? The user might want the target server to be
available instead of FATALly shutting down. This is the exact problem
the proposed patch is trying to solve.

With the GUC proposed, the user can choose what to do in these
scenarios. The user will be fully aware what she needs when she choose
to set the new GUC recovery_end_before_target_action to 'promote'
instead of default 'shutdown'.

> It look like what you actually want is some kind of a target window,
> but the window you currently propose is a hardcoded (consistency,
> given target], and it seems too dangerous to be useful.

As I said earlier, the behaviour is not too dangerous as it is not
something new that the patch is proposing, it exists today in v12 and
below. In fact, it gives a way out of a "dangerous situation" if the
user ever gets stuck in it without wasting recovery cycles and compute
resources, by quickly getting the database to be available(of course,
the responsibility lies with the user to deal with the missing WAL
files).

Regards,
Bharath Rupireddy.

Firstly, the proposed patch adds no new behaviour as such, it just

gives the ability that is existing today on v12 and below (prior to

commit dc78866 which went into v13 and later).

I think performing PITR is the user's wish - whether the primary is

available or not, it is completely the user's choice. The user might

start the PITR, when the primary is available, thinking that it sends

all the WAL files required for achieving recovery target. But imagine

a disaster happens and the primary server crashes, say the recovery

has replayed a huge bunch of WAL records (a TB may be), and the

primary failed without sending the last one or few WAL files, should

the PITR target server be failing this case after replaying a huge

bunch of WAL records? The user might want the target server to be

available instead of FATALly shutting down. This is the exact problem

the proposed patch is trying to solve.

As I said earlier, the behaviour is not too dangerous as it is not

something new that the patch is proposing, it exists today in v12 and

below. In fact, it gives a way out of a "dangerous situation" if the

user ever gets stuck in it without wasting recovery cycles and compute

resources, by quickly getting the database to be available(of course,

the responsibility lies with the user to deal with the missing WAL

files).

On Sat, Nov 13, 2021 at 6:45 PM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
>
> Firstly, the proposed patch adds no new behaviour as such, it just
> gives the ability that is existing today on v12 and below (prior to
> commit dc78866 which went into v13 and later).
>
> I think performing PITR is the user's wish - whether the primary is
> available or not, it is completely the user's choice. The user might
> start the PITR, when the primary is available, thinking that it sends
> all the WAL files required for achieving recovery target. But imagine
> a disaster happens and the primary server crashes, say the recovery
> has replayed a huge bunch of WAL records (a TB may be), and the
> primary failed without sending the last one or few WAL files, should
> the PITR target server be failing this case after replaying a huge
> bunch of WAL records? The user might want the target server to be
> available instead of FATALly shutting down. This is the exact problem
> the proposed patch is trying to solve.
>
> With the GUC proposed, the user can choose what to do in these
> scenarios. The user will be fully aware what she needs when she choose
> to set the new GUC recovery_end_before_target_action to 'promote'
> instead of default 'shutdown'.

Hi Hackers, with a recent bug report [1] in pgsql-bugs, I'm checking
if the proposal here in this thread interests anyone.

[1] https://www.postgresql.org/message-id/CALj2ACVnCsNyJTG_75%2B5Us2evfsLYz5CEhmCV4qH%3DVPa0kWOvw%40mail.gmail.com

Regards,
Bharath Rupireddy.