Thread: Too strict check when starting from a basebackup taken off a standby
Hi,
A customer recently reported getting "backup_label contains data
inconsistent with control file" after taking a basebackup from a standby
and starting it with a typo in primary_conninfo.
When starting postgres from a basebackup StartupXLOG() has the follow
code to deal with backup labels: if (haveBackupLabel) { ControlFile->backupStartPoint =
checkPoint.redo; ControlFile->backupEndRequired = backupEndRequired;
if (backupFromStandby) { if (dbstate_at_startup != DB_IN_ARCHIVE_RECOVERY)
ereport(FATAL, (errmsg("backup_label contains data inconsistent with control file"),
errhint("This means that the backup is corrupted and you will "
"haveto use another backup for recovery."))); ControlFile->backupEndPoint =
ControlFile->minRecoveryPoint; } }
while I'm not enthusiastic about the error message, that bit of code
looks sane at first glance. We certainly expect the control file to
indicate we're in recovery. Since we're unlinking the backup label
shortly afterwards we'd normally not expect to hit that case after a
shutdown in recovery.
The problem is that after reading the backup label we also have to read
the corresponding checkpoing from pg_xlog. If primary_conninfo and/or
restore_command are misconfigured and can't restore files that can only
be fixed by shutting down the cluster and fixing up recovery.conf -
which sets DB_SHUTDOWNED_IN_RECOVERY in the control file.
The easiest solution seems to be to simply also allow that as a state in
the above check. It might be nicer to not allow a ShutdownXLOG to modify
the control file et al at that stage, but I think that'd end up being
more invasive.
A short search shows that that also looks like a credible explanation
for #12128...
I plan to relax that check unless somebody comes up with a different &
better plan.
Greetings,
Andres Freund
-- Andres Freund http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training &
Services
Re: Too strict check when starting from a basebackup taken off a standby
From
Heikki Linnakangas
Date:
On 12/11/2014 05:45 AM, Andres Freund wrote:
> A customer recently reported getting "backup_label contains data
> inconsistent with control file" after taking a basebackup from a standby
> and starting it with a typo in primary_conninfo.
>
> When starting postgres from a basebackup StartupXLOG() has the follow
> code to deal with backup labels:
> if (haveBackupLabel)
> {
> ControlFile->backupStartPoint = checkPoint.redo;
> ControlFile->backupEndRequired = backupEndRequired;
>
> if (backupFromStandby)
> {
> if (dbstate_at_startup != DB_IN_ARCHIVE_RECOVERY)
> ereport(FATAL,
> (errmsg("backup_label contains data inconsistent with control file"),
> errhint("This means that the backup is corrupted and you will "
> "have to use another backup for recovery.")));
> ControlFile->backupEndPoint = ControlFile->minRecoveryPoint;
> }
> }
>
> while I'm not enthusiastic about the error message, that bit of code
> looks sane at first glance. We certainly expect the control file to
> indicate we're in recovery. Since we're unlinking the backup label
> shortly afterwards we'd normally not expect to hit that case after a
> shutdown in recovery.
Check.
> The problem is that after reading the backup label we also have to read
> the corresponding checkpoing from pg_xlog. If primary_conninfo and/or
> restore_command are misconfigured and can't restore files that can only
> be fixed by shutting down the cluster and fixing up recovery.conf -
> which sets DB_SHUTDOWNED_IN_RECOVERY in the control file.
No it doesn't. The state is set to DB_SHUTDOWNED_IN_RECOVERY in
CreateRestartPoint(). If you shut down the server before it has even
read the initial checkpoint record, it will not attempt to create a
restartpoint nor update the control file.
> The easiest solution seems to be to simply also allow that as a state in
> the above check. It might be nicer to not allow a ShutdownXLOG to modify
> the control file et al at that stage, but I think that'd end up being
> more invasive.
>
> A short search shows that that also looks like a credible explanation
> for #12128...
Yeah. I was not able to reproduce this, but I'm clearly missing
something, since both you and Sergey have seen this happening. Can you
write a script to reproduce?
- Heikki
On December 11, 2014 9:56:09 AM CET, Heikki Linnakangas <hlinnakangas@vmware.com> wrote:
>On 12/11/2014 05:45 AM, Andres Freund wrote:
>> A customer recently reported getting "backup_label contains data
>> inconsistent with control file" after taking a basebackup from a
>standby
>> and starting it with a typo in primary_conninfo.
>>
>> When starting postgres from a basebackup StartupXLOG() has the follow
>> code to deal with backup labels:
>> if (haveBackupLabel)
>> {
>> ControlFile->backupStartPoint = checkPoint.redo;
>> ControlFile->backupEndRequired = backupEndRequired;
>>
>> if (backupFromStandby)
>> {
>> if (dbstate_at_startup != DB_IN_ARCHIVE_RECOVERY)
>> ereport(FATAL,
>> (errmsg("backup_label contains data
>inconsistent with control file"),
>> errhint("This means that the backup is
>corrupted and you will "
>> "have to use another backup for
>recovery.")));
>> ControlFile->backupEndPoint =
>ControlFile->minRecoveryPoint;
>> }
>> }
>>
>> while I'm not enthusiastic about the error message, that bit of code
>> looks sane at first glance. We certainly expect the control file to
>> indicate we're in recovery. Since we're unlinking the backup label
>> shortly afterwards we'd normally not expect to hit that case after a
>> shutdown in recovery.
>
>Check.
>
>> The problem is that after reading the backup label we also have to
>read
>> the corresponding checkpoing from pg_xlog. If primary_conninfo and/or
>> restore_command are misconfigured and can't restore files that can
>only
>> be fixed by shutting down the cluster and fixing up recovery.conf -
>> which sets DB_SHUTDOWNED_IN_RECOVERY in the control file.
>
>No it doesn't. The state is set to DB_SHUTDOWNED_IN_RECOVERY in
>CreateRestartPoint(). If you shut down the server before it has even
>read the initial checkpoint record, it will not attempt to create a
>restartpoint nor update the control file.
Yes, it does. There's a shortcut that just sets the state in the control file and then exits.
>> The easiest solution seems to be to simply also allow that as a state
>in
>> the above check. It might be nicer to not allow a ShutdownXLOG to
>modify
>> the control file et al at that stage, but I think that'd end up being
>> more invasive.
>>
>> A short search shows that that also looks like a credible explanation
>> for #12128...
>
>Yeah. I was not able to reproduce this, but I'm clearly missing
>something, since both you and Sergey have seen this happening. Can you
>write a script to reproduce?
Not right now, I only have my mobile... Its quite easy though. Create a pg-basebackup from a standby. Create a
recovery.confwith a broken primary conninfo. Start. Shutdown. Fix conninfo. Start.
Andres
--
Please excuse brevity and formatting - I am writing this on my mobile phone.
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
Re: Too strict check when starting from a basebackup taken off a standby
From
Marco Nenciarini
Date:
Il 11/12/14 12:38, Andres Freund ha scritto: > On December 11, 2014 9:56:09 AM CET, Heikki Linnakangas <hlinnakangas@vmware.com> wrote: >> On 12/11/2014 05:45 AM, Andres Freund wrote: >> >> Yeah. I was not able to reproduce this, but I'm clearly missing >> something, since both you and Sergey have seen this happening. Can you >> write a script to reproduce? > > Not right now, I only have my mobile... Its quite easy though. Create a pg-basebackup from a standby. Create a recovery.confwith a broken primary conninfo. Start. Shutdown. Fix conninfo. Start. > Just tested it. There steps are not sufficient to reproduce the issue on a test installation. I suppose because, on small test datadir, the checkpoint location and the redo location on the pg_control are the same present in the backup_label. To trigger this bug you need to have at least a restartpoint happened on standby between the start and the end of the backup. you could simulate it issuing a checkpoint on master, a checkpoint on standby (to force a restartpoint), then copying the pg_control from the standby. This way I've been able to reproduce it. Regards, Marco -- Marco Nenciarini - 2ndQuadrant Italy PostgreSQL Training, Services and Support marco.nenciarini@2ndQuadrant.it | www.2ndQuadrant.it
Re: Too strict check when starting from a basebackup taken off a standby
From
Heikki Linnakangas
Date:
On 12/11/2014 04:21 PM, Marco Nenciarini wrote: > Il 11/12/14 12:38, Andres Freund ha scritto: >> On December 11, 2014 9:56:09 AM CET, Heikki Linnakangas <hlinnakangas@vmware.com> wrote: >>> On 12/11/2014 05:45 AM, Andres Freund wrote: >>> >>> Yeah. I was not able to reproduce this, but I'm clearly missing >>> something, since both you and Sergey have seen this happening. Can you >>> write a script to reproduce? >> >> Not right now, I only have my mobile... Its quite easy though. Create a pg-basebackup from a standby. Create a recovery.confwith a broken primary conninfo. Start. Shutdown. Fix conninfo. Start. >> > > Just tested it. There steps are not sufficient to reproduce the issue on > a test installation. I suppose because, on small test datadir, the > checkpoint location and the redo location on the pg_control are the same > present in the backup_label. > > To trigger this bug you need to have at least a restartpoint happened on > standby between the start and the end of the backup. > > you could simulate it issuing a checkpoint on master, a checkpoint on > standby (to force a restartpoint), then copying the pg_control from the > standby. > > This way I've been able to reproduce it. Ok, got it. I was able to reproduce this by using pg_basebackup --max-rate=1024, and issuing "CHECKPOINT" in the standby while the backup was running. Yeah, I agree with Andres' plan to relax the check, to also allow DB_SHUTDOWNED_IN_RECOVERY state. - Heikki
On 2014-12-16 18:37:48 +0200, Heikki Linnakangas wrote: > On 12/11/2014 04:21 PM, Marco Nenciarini wrote: > >Il 11/12/14 12:38, Andres Freund ha scritto: > >>On December 11, 2014 9:56:09 AM CET, Heikki Linnakangas <hlinnakangas@vmware.com> wrote: > >>>On 12/11/2014 05:45 AM, Andres Freund wrote: > >>> > >>>Yeah. I was not able to reproduce this, but I'm clearly missing > >>>something, since both you and Sergey have seen this happening. Can you > >>>write a script to reproduce? > >> > >>Not right now, I only have my mobile... Its quite easy though. Create a pg-basebackup from a standby. Create a recovery.confwith a broken primary conninfo. Start. Shutdown. Fix conninfo. Start. > >> > > > >Just tested it. There steps are not sufficient to reproduce the issue on > >a test installation. I suppose because, on small test datadir, the > >checkpoint location and the redo location on the pg_control are the same > >present in the backup_label. > > > >To trigger this bug you need to have at least a restartpoint happened on > >standby between the start and the end of the backup. > > > >you could simulate it issuing a checkpoint on master, a checkpoint on > >standby (to force a restartpoint), then copying the pg_control from the > >standby. > > > >This way I've been able to reproduce it. > > Ok, got it. I was able to reproduce this by using pg_basebackup > --max-rate=1024, and issuing "CHECKPOINT" in the standby while the backup > was running. FWIW, I can reproduce it without any such hangups. I've just tested it using my local scripts: # create primary $ reinit-pg-dev-master $ run-pg-dev-master # create first standby $ reinit-pg-dev-master-standby $ run-pg-dev-master-standby # create 2nd standby $ pg_basebackup -h /tmp/ -p 5441 -D /tmp/tree --write-recovery-conf $ PGHOST=frakbar run-pg-dev-master-standby -D /tmp/tree LOG: creating missing WAL directory "pg_xlog/archive_status" LOG: entering standby mode FATAL: could not connect to the primary server: could not translate host name "frakbar" to address: Name or service notknown $ PGHOST=/tmp run-pg-dev-master-standby -D /tmp/tree LOG: started streaming WAL from primary at 0/2000000 on timeline 1 FATAL: backup_label contains data inconsistent with control file HINT: This means that the backup is corrupted and you will have to use another backup for recovery. After the fix I just pushed that sequence works. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services