Re: [GENERAL] Shared WAL archive between master and standby: WALs notalways identical - Mailing list pgsql-general

From Sasa Vilic
Subject Re: [GENERAL] Shared WAL archive between master and standby: WALs notalways identical
Date
Msg-id CAOJhpYfG7NrGML6UpHUHBth6MBGUUWj6h=xuywi0Hvvg9amR+w@mail.gmail.com
Whole thread Raw
In response to Re: [GENERAL] Shared WAL archive between master and standby: WALs notalways identical  (Sasa Vilic <sasavilic@gmail.com>)
Responses Re: [GENERAL] Shared WAL archive between master and standby: WALs notalways identical  ("David G. Johnston" <david.g.johnston@gmail.com>)
List pgsql-general
And also this:

"""
If archive_mode is set to on, the archiver is not enabled during recovery or standby mode. If the standby server is promoted, it will start archiving after the promotion, but will not archive any WAL it did not generate itself. To get a complete series of WAL files in the archive, you must ensure that all WAL is archived, before it reaches the standby
"""

If I understand it correctly, WAL started on master but finished on standby (after failover) will still not be (old master is presumably dead and new master must wait for next WAL segment). Of course, the next WAL segment will be sent by new master, but we are going to miss exactly this one WAL segment during which failover occurred and thus introduce interruption in our WAL stream. Am I right?

Regards,
Sasa

On 28 February 2017 at 02:33, Sasa Vilic <sasavilic@gmail.com> wrote:
Hi David,

thanks for the answer. I read this in documentation but here there is a corner case that I am not sure how to handle:
"""
This requires more care in the archive_command, as it must be careful to not overwrite an existing file with different contents, but return success if the exactly same file is archived twice.
"""
But what I am supposed to do when content differs? Still return success and ignore or return error? If I return error, wouldn't that prevent wal archiver slave from pushing further WALs?

Regards,
Sasa


On 28 February 2017 at 02:10, David G. Johnston <david.g.johnston@gmail.com> wrote:
On Mon, Feb 27, 2017 at 5:40 PM, Sasa Vilic <sasavilic@gmail.com> wrote:
Aren't WALs from master and standby supposed to be identical?

​This would seem unwise to assume on its face and at least one piece of documentation directly mentions that it is false:


"""
When continuous WAL archiving is used in a standby, there are two different scenarios: the WAL archive can be shared between the primary and the standby, or the standby can have its own WAL archive. When the standby has its own WAL archive, set archive_mode to always, and the standby will call the archive command for every WAL segment it receives, whether it's by restoring from the archive or by streaming replication. The shared archive can be handled similarly, but the archive_command must test if the file being archived exists already, and if the existing file has identical contents. This requires more care in the archive_command, as it must be careful to not overwrite an existing file with different contents, but return success if the exactly same file is archived twice. And all that must be done free of race conditions, if two servers attempt to archive the same file at the same time.
"""

​The contents of both must match with respect to the data files but there are likely things that go into the master WAL stream solely for the purpose of communicating with a standby - ​and possibly some standby concepts that would be unique to the standby's WAL - that would cause them to differ.  Not familiar enough to quickly list examples of what those might be.  But IIUC the system seems designed around master->slave replication and doesn't support slave daisy-chains.

David J.



pgsql-general by date:

Previous
From: Sasa Vilic
Date:
Subject: Re: [GENERAL] Shared WAL archive between master and standby: WALs notalways identical
Next
From: "David G. Johnston"
Date:
Subject: Re: [GENERAL] Shared WAL archive between master and standby: WALs notalways identical