Re: Critical failure of standby - Mailing list pgsql-general

From James Sewell
Subject Re: Critical failure of standby
Date
Msg-id CAANVwEsmzXgSBq=h7krw-bXBwdVgY9xBHDqDFyYCar1o5159ZA@mail.gmail.com
Whole thread Raw
In response to Re: Critical failure of standby  (Jeff Janes <jeff.janes@gmail.com>)
Responses Re: Critical failure of standby  (John R Pierce <pierce@hogranch.com>)
Re: Critical failure of standby  (Jeff Janes <jeff.janes@gmail.com>)
List pgsql-general
Those are all good questions.

Essentially this is a situation where DR is network separated from Prod - so I would expect the archive command to fail. I'll have to check the script it must not be passing the error back through to PostgreSQL.

This still shouldn't cause database corruption though right? - it's just not getting WALs.

Cheers,

James Sewell,
PostgreSQL Team Lead / Solutions Architect 

 

Suite 112, Jones Bay Wharf, 26-32 Pirrama Road, Pyrmont NSW 2009

On Tue, Aug 16, 2016 at 2:09 AM, Jeff Janes <jeff.janes@gmail.com> wrote:
On Thu, Aug 11, 2016 at 10:39 PM, James Sewell <james.sewell@jirotech.com> wrote:
Hello,

We recently experienced a critical failure when failing to a DR environment.

This is in the following environment:

  • 3 x PostgreSQL machines in Prod in a sync replication cluster
  • 3 x PostgreSQL machines in DR, with a single machine async and the other two cascading from the first machine.
There was network failure which isolated Production from everything else, Production has no errors during this time (and has now come back OK).

DR did not tolerate the break, the following appeared in the logs and none of them can start postgres. There were no queries coming into DR at the time of the break.

Please note that the "Host Key verification failed" messages are due to the scp command not functioning. This means restore_command is not working to restore from the XLOG archive, but should not effect anything else.


In my experience, PostgreSQL issues its own error messages when restore_command fails.  So I see both the error from the command itself, and an error from PostgreSQL.  Why don't you see that?  Is the restore_command failing, but then reporting that it succeeded?  

And if you can't get files from the XLOG archive, why do you think that that is OK?

Cheers,

Jeff



The contents of this email are confidential and may be subject to legal or professional privilege and copyright. No representation is made that this email is free of viruses or other defects. If you have received this communication in error, you may not copy or distribute any part of it or otherwise disclose its contents to anyone. Please advise the sender of your incorrect receipt of this correspondence.

pgsql-general by date:

Previous
From: Michael Paquier
Date:
Subject: Re: pgbasebackup is failing after truncate
Next
From: John R Pierce
Date:
Subject: Re: Critical failure of standby