Thread: Archiving control (a part of synch rep patches)
Hi, Some people want to disable xlog archiving only during synchronous replication. So, first, I created and attached the self-contained patch to control xlog archiving dynamically. Since it would still take time to fix whole patch of synch rep, and opening it at a time would burden reviewers, I opened this patch this time. http://archives.postgresql.org/pgsql-hackers/2008-12/msg00718.php This patch provides three abilities: (1) Turn off xlog archiving from the specified file. For example, if the xlog file including replication starting position is specified, we can stop archiving the xlog files which are generated during replication. Specifically, we only skip executing archive command after turning off, that is, archiver is still in progress and .done file is created. This reduces the burden of resuming archiving. (2) Turn back on xlog archiving from the specified file. (3) Revert the arhive status file of the specified file from .done to .ready. This would try to archive the corresponding xlog file again. The archive status of only skipped xlog file can be reverted. We resume xlog archiving by using (2) and (3), when replication ends. (1) and (2) are provided as PgArchControl() in pgarch.c. (3) is provided as XLogArchiveRevert() in xlog.c. Please feel free to comment! Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center
Attachment
Hi, On Wed, Dec 24, 2008 at 4:39 PM, Fujii Masao <masao.fujii@gmail.com> wrote: > (3) > Revert the arhive status file of the specified file from .done to .ready. > This would try to archive the corresponding xlog file again. The archive > status of only skipped xlog file can be reverted. > > We resume xlog archiving by using (2) and (3), when replication > ends. Umm.. reverting is very tricky, then it might become hotbeds of bugs. So, I will reconsider the method of resuming archive. http://archives.postgresql.org/pgsql-hackers/2008-12/msg00723.php > That conceivably might mean we need > to write a .ready message after a WAL file filled, which might mean we > have problems if the replication timeout is longer than the checkpoint > timeout, but that seems an unlikely configuration. And if anybody has a > problem with that we just recommend they use the "always" mode. It's also problem that "reverting" competes with xlog deletion by checkpoint. Perhaps we should create .ready after the xlog file fills and it's replicated during replication. I will consider this course deeply. Happy Holidays! -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center
On Wed, 2008-12-24 at 19:04 +0900, Fujii Masao wrote: > Hi, > > On Wed, Dec 24, 2008 at 4:39 PM, Fujii Masao <masao.fujii@gmail.com> wrote: > > (3) > > Revert the arhive status file of the specified file from .done to .ready. > > This would try to archive the corresponding xlog file again. The archive > > status of only skipped xlog file can be reverted. > > > > We resume xlog archiving by using (2) and (3), when replication > > ends. > > Umm.. reverting is very tricky, then it might become hotbeds of bugs. > So, I will reconsider the method of resuming archive. Yeh, don't do that. No worries if we send a few files too many. Tears if we miss just one. * Turn it off for the *next* file to be archived. Keep track of last file sent. * Turn it on again, mark *all* WAL files as .ready in sequence low to high, as long as they are higher than the last file sent. -- Simon Riggs www.2ndQuadrant.comPostgreSQL Training, Services and Support
Hi, On Wed, Dec 24, 2008 at 7:26 PM, Simon Riggs <simon@2ndquadrant.com> wrote: > > On Wed, 2008-12-24 at 19:04 +0900, Fujii Masao wrote: >> Hi, >> >> On Wed, Dec 24, 2008 at 4:39 PM, Fujii Masao <masao.fujii@gmail.com> wrote: >> > (3) >> > Revert the arhive status file of the specified file from .done to .ready. >> > This would try to archive the corresponding xlog file again. The archive >> > status of only skipped xlog file can be reverted. >> > >> > We resume xlog archiving by using (2) and (3), when replication >> > ends. >> >> Umm.. reverting is very tricky, then it might become hotbeds of bugs. >> So, I will reconsider the method of resuming archive. > > Yeh, don't do that. > > No worries if we send a few files too many. Tears if we miss just one. > > * Turn it off for the *next* file to be archived. Keep track of last > file sent. OK > * Turn it on again, mark *all* WAL files as .ready in sequence low to > high, as long as they are higher than the last file sent. You mean as follows? - we don't create .ready during replication. - we create .ready of the xlog files which are higher than the last sent when replication ends, and forbid that subsequentcheckpoint create .ready of lower xlog files. Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center