Thread: Docs for archive_cleanup_command are poor
Hi folks, I have just set up HS+SR for the first time, and for the most part, the docs were excellent. The one exception for me was the discussion of archive_cleanup_command. This is a pretty important part of constructing a healthy standby server, and IMO the docs don't give it the treatment it deserves. Under "25.2.4. Setting Up a Standby Server", we have: "You can use archive_cleanup_command to prune the archive of files no longer needed by the standby." ... then a few paragraphs later ... "If you're using a WAL archive, its size can be minimized using the archive_cleanup_command option to remove files that are no longer required by the standby server. Note however, that if you're using the archive for backup purposes, you need to retain files needed to recover from at least the latest base backup, even if they're no longer needed by the standby." So there are a couple of brief mentions of what archive_cleanup_command is for, but nothing about how it works, no exampes of how to use it, and no links at all. Contrast how we deal with archive_command, restore_command and primary_conninfo. I'd like to suggest a few ways we could improve on this: 1. Remove the former paragraph. It's stranded out there on its own in the middle of some unrelated text, and doesn't say anything of substance not also said in the latter paragraph. 2. Include an example archive_cleanup_command in the recovery.conf example snippet. 3. Link to 26.1 which actually explains how a_c_c works. 4. Mention, and link to, pg_archivecleanup from both 25.2.4 and 26.1. This is the utility that most newcomers to WAL archiving will want to use, so it's rather weird of us not to advertise it. I'm willing to write a patch for this, but I thought I'd raise the suggestions on-list first, before getting too invested. So, please comment if you have an opinion on this. Cheers, BJ
On Sat, Oct 9, 2010 at 10:04 AM, Brendan Jurd <direvus@gmail.com> wrote: > Hi folks, > > I have just set up HS+SR for the first time, and for the most part, > the docs were excellent. The one exception for me was the discussion > of archive_cleanup_command. This is a pretty important part of > constructing a healthy standby server, and IMO the docs don't give it > the treatment it deserves. > > Under "25.2.4. Setting Up a Standby Server", we have: > > "You can use archive_cleanup_command to prune the archive of files no > longer needed by the standby." > > ... then a few paragraphs later ... > > "If you're using a WAL archive, its size can be minimized using the > archive_cleanup_command option to remove files that are no longer > required by the standby server. Note however, that if you're using the > archive for backup purposes, you need to retain files needed to > recover from at least the latest base backup, even if they're no > longer needed by the standby." > > So there are a couple of brief mentions of what > archive_cleanup_command is for, but nothing about how it works, no > exampes of how to use it, and no links at all. Contrast how we deal > with archive_command, restore_command and primary_conninfo. > > I'd like to suggest a few ways we could improve on this: > > 1. Remove the former paragraph. It's stranded out there on its own in > the middle of some unrelated text, and doesn't say anything of > substance not also said in the latter paragraph. > > 2. Include an example archive_cleanup_command in the recovery.conf > example snippet. > > 3. Link to 26.1 which actually explains how a_c_c works. > > 4. Mention, and link to, pg_archivecleanup from both 25.2.4 and 26.1. > This is the utility that most newcomers to WAL archiving will want to > use, so it's rather weird of us not to advertise it. > > I'm willing to write a patch for this, but I thought I'd raise the > suggestions on-list first, before getting too invested. So, please > comment if you have an opinion on this. Agreed. And, ISTM that we should mention that we must not just specify pg_archivecleanup in archive_cleanup_command when there are multiple standby servers. This is because, in that case, we must calculate the oldest restart point in those standbys and delete the archived WAL files according to that point. Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center
On Tue, Oct 12, 2010 at 8:28 AM, Fujii Masao <masao.fujii@gmail.com> wrote: > On Sat, Oct 9, 2010 at 10:04 AM, Brendan Jurd <direvus@gmail.com> wrote: >> Hi folks, >> >> I have just set up HS+SR for the first time, and for the most part, >> the docs were excellent. The one exception for me was the discussion >> of archive_cleanup_command. This is a pretty important part of >> constructing a healthy standby server, and IMO the docs don't give it >> the treatment it deserves. >> >> Under "25.2.4. Setting Up a Standby Server", we have: >> >> "You can use archive_cleanup_command to prune the archive of files no >> longer needed by the standby." >> >> ... then a few paragraphs later ... >> >> "If you're using a WAL archive, its size can be minimized using the >> archive_cleanup_command option to remove files that are no longer >> required by the standby server. Note however, that if you're using the >> archive for backup purposes, you need to retain files needed to >> recover from at least the latest base backup, even if they're no >> longer needed by the standby." >> >> So there are a couple of brief mentions of what >> archive_cleanup_command is for, but nothing about how it works, no >> exampes of how to use it, and no links at all. Contrast how we deal >> with archive_command, restore_command and primary_conninfo. >> >> I'd like to suggest a few ways we could improve on this: >> >> 1. Remove the former paragraph. It's stranded out there on its own in >> the middle of some unrelated text, and doesn't say anything of >> substance not also said in the latter paragraph. >> >> 2. Include an example archive_cleanup_command in the recovery.conf >> example snippet. >> >> 3. Link to 26.1 which actually explains how a_c_c works. >> >> 4. Mention, and link to, pg_archivecleanup from both 25.2.4 and 26.1. >> This is the utility that most newcomers to WAL archiving will want to >> use, so it's rather weird of us not to advertise it. >> >> I'm willing to write a patch for this, but I thought I'd raise the >> suggestions on-list first, before getting too invested. So, please >> comment if you have an opinion on this. > > Agreed. Is someone working on a patch? > And, ISTM that we should mention that we must not just specify > pg_archivecleanup in archive_cleanup_command when there are multiple > standby servers. This is because, in that case, we must calculate > the oldest restart point in those standbys and delete the archived > WAL files according to that point. How do we expect people to do that, by the way? -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
On 14 October 2010 08:45, Robert Haas <robertmhaas@gmail.com> wrote: > Is someone working on a patch? Yes, I will prepare a patch to get us started. Cheers, BJ
On Fri, 2010-10-15 at 02:24 +1100, Brendan Jurd wrote: > I'll drop this onto the next open commitfest. If it passes muster, it > sure wouldn't hurt to backpatch it to 9.0. Committed. Not sure there's anything there worth backpatching? There aren't any doc bugs there. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Training and Services
On 12 October 2010 23:28, Fujii Masao <masao.fujii@gmail.com> wrote: > On Sat, Oct 9, 2010 at 10:04 AM, Brendan Jurd <direvus@gmail.com> wrote: >> I have just set up HS+SR for the first time, and for the most part, >> the docs were excellent. The one exception for me was the discussion >> of archive_cleanup_command. This is a pretty important part of >> constructing a healthy standby server, and IMO the docs don't give it >> the treatment it deserves. ... > Agreed. > > And, ISTM that we should mention that we must not just specify > pg_archivecleanup in archive_cleanup_command when there are multiple > standby servers. This is because, in that case, we must calculate > the oldest restart point in those standbys and delete the archived > WAL files according to that point. As promised, here is a patch to try to address $SUBJECT. Summary of changes: In 25.2.4. "Setting Up a Standby Server": * Get rid of the extraneous short paragraph, * move the full-size paragraph up to where the now-extinct short para was, * add an archive_cleanup_command to the example recovery.conf, * flesh out the wording, * add links to 26.1 and F.22. In 26.1. "Archive recovery settings": * Add detail to the description of how it works, * add an example recovery.conf snippet, * per Fujii-san's comment, indicate that multi-standby setups require more finesse, * link to F.22. In F.22. "pg_archivecleanup": * Edit and clarify wording, * standardise label for the <archivelocation> argument, * again indicate the multi-standby issue, * link to 25.2. I'll drop this onto the next open commitfest. If it passes muster, it sure wouldn't hurt to backpatch it to 9.0. Cheers, BJ
Attachment
On Thu, Oct 14, 2010 at 2:33 PM, Simon Riggs <simon@2ndquadrant.com> wrote: > On Fri, 2010-10-15 at 02:24 +1100, Brendan Jurd wrote: > >> I'll drop this onto the next open commitfest. If it passes muster, it >> sure wouldn't hurt to backpatch it to 9.0. > > Committed. Not sure there's anything there worth backpatching? There > aren't any doc bugs there. It's a while until 9.1 comes out, so it might be helpful to back-patch to 9.0. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
On 15 October 2010 05:33, Simon Riggs <simon@2ndquadrant.com> wrote: > On Fri, 2010-10-15 at 02:24 +1100, Brendan Jurd wrote: >> I'll drop this onto the next open commitfest. If it passes muster, it >> sure wouldn't hurt to backpatch it to 9.0. > > Committed. Not sure there's anything there worth backpatching? There > aren't any doc bugs there. > Thanks for the commit Simon. Agreed that there are no doc bugs. The reason I suggested a backpatch is that I'm concerned that a lot of people are going to be approaching the whole Standby topic for the first time with 9.0, so it would be nice to give those folks an accessible account of how archive_cleanup_command is meant to be used. I was also working from the assumption that the "we only packpatch bug fixes" policy applied to the code, not so much to the documentation. If I was in error about that, well fair enough then. Cheers, BJ
Brendan Jurd <direvus@gmail.com> writes:
> Agreed that there are no doc bugs. The reason I suggested a backpatch
> is that I'm concerned that a lot of people are going to be approaching
> the whole Standby topic for the first time with 9.0, so it would be
> nice to give those folks an accessible account of how
> archive_cleanup_command is meant to be used.
Yeah, if this is a material improvement in the usefulness of the HS
docs, I think including it into 9.0.x isn't a bad idea.
regards, tom lane
On Thu, 2010-10-14 at 18:05 -0400, Tom Lane wrote: > Brendan Jurd <direvus@gmail.com> writes: > > Agreed that there are no doc bugs. The reason I suggested a backpatch > > is that I'm concerned that a lot of people are going to be approaching > > the whole Standby topic for the first time with 9.0, so it would be > > nice to give those folks an accessible account of how > > archive_cleanup_command is meant to be used. > > Yeah, if this is a material improvement in the usefulness of the HS > docs, I think including it into 9.0.x isn't a bad idea. Done. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Training and Services