Thread: BUG #6528: pglesslog still referenced in docs, but no 9.1 support
BUG #6528: pglesslog still referenced in docs, but no 9.1 support
From
stuart.bishop@canonical.com
Date:
The following bug has been logged on the website: Bug reference: 6528 Logged by: Stuart Bishop Email address: stuart.bishop@canonical.com PostgreSQL version: 9.1.3 Operating system: Ubuntu Description:=20=20=20=20=20=20=20=20 In section 24.3.5.1 of the docs on setting up continuous archiving, there are a few paragraphs and examples for using pglesslog (pg_compresslog and pg_decompresslog). This tool did not get out of beta for 9.0 and the only reports I find for 9.1 are for not being able to compile it. Suggest removing it from the docs unless someone wants to pull this tool into the core.
On Tue, Mar 13, 2012 at 5:00 AM, <stuart.bishop@canonical.com> wrote: > The following bug has been logged on the website: > > Bug reference: =A0 =A0 =A06528 > Logged by: =A0 =A0 =A0 =A0 =A0Stuart Bishop > Email address: =A0 =A0 =A0stuart.bishop@canonical.com > PostgreSQL version: 9.1.3 > Operating system: =A0 Ubuntu > Description: > > In section 24.3.5.1 of the docs on setting up continuous archiving, there > are a few paragraphs and examples for using pglesslog (pg_compresslog and > pg_decompresslog). This tool did not get out of beta for 9.0 and the only > reports I find for 9.1 are for not being able to compile it. Suggest > removing it from the docs unless someone wants to pull this tool into the > core. This complaint appears to be accurate. I think we should go ahead and remove that mention. --=20 Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
On Mon, Apr 09, 2012 at 10:10:34AM -0400, Robert Haas wrote: > On Tue, Mar 13, 2012 at 5:00 AM, <stuart.bishop@canonical.com> wrote: > > The following bug has been logged on the website: > > > > Bug reference: 6528 > > Logged by: Stuart Bishop > > Email address: stuart.bishop@canonical.com > > PostgreSQL version: 9.1.3 > > Operating system: Ubuntu > > Description: > > > > In section 24.3.5.1 of the docs on setting up continuous archiving, there > > are a few paragraphs and examples for using pglesslog (pg_compresslog and > > pg_decompresslog). This tool did not get out of beta for 9.0 and the only > > reports I find for 9.1 are for not being able to compile it. Suggest > > removing it from the docs unless someone wants to pull this tool into the > > core. > > This complaint appears to be accurate. I think we should go ahead and > remove that mention. Agreed; removed with the attached patch. I didn't bother keeping the gzip mention because I assume there is little value to using without pglesslog. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. +
Attachment
Bruce Momjian <bruce@momjian.us> wrote: > Robert Haas wrote: >> <stuart.bishop@canonical.com> wrote: >>> In section 24.3.5.1 of the docs on setting up continuous >>> archiving, there are a few paragraphs and examples for using >>> pglesslog (pg_compresslog and pg_decompresslog). This tool did >>> not get out of beta for 9.0 and the only reports I find for 9.1 >>> are for not being able to compile it. Suggest removing it from >>> the docs unless someone wants to pull this tool into the core. >> >> This complaint appears to be accurate. I think we should go >> ahead and remove that mention. > > Agreed; removed with the attached patch. I didn't bother keeping > the gzip mention because I assume there is little value to using > without pglesslog. We found that gzip without doing something about the unused portion of the WAL file generally reduced file size by about 50%, so it still might be worth mentioning that. -Kevin
Excerpts from Bruce Momjian's message of lun abr 09 15:13:10 -0300 2012: > On Mon, Apr 09, 2012 at 10:10:34AM -0400, Robert Haas wrote: > > This complaint appears to be accurate. I think we should go ahead and > > remove that mention. >=20 > Agreed; removed with the attached patch. I didn't bother keeping the > gzip mention because I assume there is little value to using without > pglesslog. I'm not sure that assumption holds. I think you're thinking of the hack that zeroes out the "empty" holes in the middle of data pages; those didn't compress well unless zeroed out before compression. This tool is not about that, but rather about removing redundant info from WAL files. It seems to me that WAL files would be as gzip-compressible regardless of pglesslog being applied. (Another related tool is clearxlogtail which zeroes areas from WAL files when they are empty because of an early switch due to archive timeout). The funny thing is, apparently pg_lesslog was intentionally broken by changing XLR_BKP_REMOVABLE to XLP_BKP_REMOVABLE (different semantics?) and the fixes to make it compile again look simple. It's a bit strange that NTT stopped maintaining the tool .. Maybe it wasn't useful anymore? --=20 =C3=81lvaro Herrera <alvherre@commandprompt.com> The PostgreSQL Company - Command Prompt, Inc. PostgreSQL Replication, Consulting, Custom Development, 24x7 support
On Mon, Apr 09, 2012 at 03:37:09PM -0300, Alvaro Herrera wrote: > > Excerpts from Bruce Momjian's message of lun abr 09 15:13:10 -0300 2012: > > On Mon, Apr 09, 2012 at 10:10:34AM -0400, Robert Haas wrote: > > > > This complaint appears to be accurate. I think we should go ahead and > > > remove that mention. > > > > Agreed; removed with the attached patch. I didn't bother keeping the > > gzip mention because I assume there is little value to using without > > pglesslog. > > I'm not sure that assumption holds. I think you're thinking of the hack > that zeroes out the "empty" holes in the middle of data pages; those > didn't compress well unless zeroed out before compression. This tool is > not about that, but rather about removing redundant info from WAL files. > It seems to me that WAL files would be as gzip-compressible regardless > of pglesslog being applied. (Another related tool is clearxlogtail OK, docs updated with the attached patch. > which zeroes areas from WAL files when they are empty because of an > early switch due to archive timeout). Should we document that? > The funny thing is, apparently pg_lesslog was intentionally broken by > changing XLR_BKP_REMOVABLE to XLP_BKP_REMOVABLE (different semantics?) > and the fixes to make it compile again look simple. It's a bit strange > that NTT stopped maintaining the tool .. Maybe it wasn't useful anymore? No idea. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. +
Attachment
Bruce Momjian <bruce@momjian.us> wrote: > On Mon, Apr 09, 2012 at 03:37:09PM -0300, Alvaro Herrera wrote: >> (Another related tool is clearxlogtail which zeroes areas from >> WAL files when they are empty because of an early switch due to >> archive timeout). > > Should we document that? Our shop has been using that since before pglesslog existed, and it has continued to work across many major releases with no change to source code because it doesn't get down to the level of looking at the xlog records themselves, just the segment and page-level structures. I should probably put packaging that up as a proper extension and posting it to PGXN as another thing on the list of things I'll do when I stumble over that pot of round tuits at the end of the rainbow. Right now the source is on pgfoundry. If anybody wants to do anything with it before I find time, feel free. -Kevin
On Mon, Apr 9, 2012 at 02:07:43PM -0500, Kevin Grittner wrote:
> Bruce Momjian <bruce@momjian.us> wrote:
> > On Mon, Apr 09, 2012 at 03:37:09PM -0300, Alvaro Herrera wrote:
>
> >> (Another related tool is clearxlogtail which zeroes areas from
> >> WAL files when they are empty because of an early switch due to
> >> archive timeout).
> >
> > Should we document that?
>
> Our shop has been using that since before pglesslog existed, and it
> has continued to work across many major releases with no change to
> source code because it doesn't get down to the level of looking at
> the xlog records themselves, just the segment and page-level
> structures. I should probably put packaging that up as a proper
> extension and posting it to PGXN as another thing on the list of
> things I'll do when I stumble over that pot of round tuits at the
> end of the rainbow. Right now the source is on pgfoundry. If
> anybody wants to do anything with it before I find time, feel free.
I was going to add a mention of this to our docs, but it seems there are
no released files for the project:
http://pgfoundry.org/frs/?group_id=1000308
--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com
+ It's impossible for everything to be true. +
Bruce Momjian <bruce@momjian.us> wrote: > On Mon, Apr 9, 2012 at 02:07:43PM -0500, Kevin Grittner wrote: >> Bruce Momjian <bruce@momjian.us> wrote: >>> On Mon, Apr 09, 2012 at 03:37:09PM -0300, Alvaro Herrera wrote: >>> >>>> (Another related tool is clearxlogtail which zeroes areas from >>>> WAL files when they are empty because of an early switch due to >>>> archive timeout). >>> >>> Should we document that? >> >> Our shop has been using that since before pglesslog existed, and >> it has continued to work across many major releases with no >> change to source code because it doesn't get down to the level of >> looking at the xlog records themselves, just the segment and >> page-level structures. I should probably put packaging that up >> as a proper extension and posting it to PGXN as another thing on >> the list of things I'll do when I stumble over that pot of round >> tuits at the end of the rainbow. Right now the source is on >> pgfoundry. If anybody wants to do anything with it before I find >> time, feel free. > > I was going to add a mention of this to our docs, but it seems > there are no released files for the project: There is a single 175 line C source file to generate a stand-alone executable filter. The source has not needed any changes since it was finished over 4 years ago. Besides that C file, there is only a make file. I wasn't sure what form of packaging would be easier than just checking out the source from the SCM. What do you think would be the right thing to do with it at this point? -Kevin
On Tue, Sep 4, 2012 at 12:11:53PM -0500, Kevin Grittner wrote: > Bruce Momjian <bruce@momjian.us> wrote: > > On Mon, Apr 9, 2012 at 02:07:43PM -0500, Kevin Grittner wrote: > >> Bruce Momjian <bruce@momjian.us> wrote: > >>> On Mon, Apr 09, 2012 at 03:37:09PM -0300, Alvaro Herrera wrote: > >>> > >>>> (Another related tool is clearxlogtail which zeroes areas from > >>>> WAL files when they are empty because of an early switch due to > >>>> archive timeout). > >>> > >>> Should we document that? > >> > >> Our shop has been using that since before pglesslog existed, and > >> it has continued to work across many major releases with no > >> change to source code because it doesn't get down to the level of > >> looking at the xlog records themselves, just the segment and > >> page-level structures. I should probably put packaging that up > >> as a proper extension and posting it to PGXN as another thing on > >> the list of things I'll do when I stumble over that pot of round > >> tuits at the end of the rainbow. Right now the source is on > >> pgfoundry. If anybody wants to do anything with it before I find > >> time, feel free. > > > > I was going to add a mention of this to our docs, but it seems > > there are no released files for the project: > > There is a single 175 line C source file to generate a stand-alone > executable filter. The source has not needed any changes since it > was finished over 4 years ago. Besides that C file, there is only a > make file. I wasn't sure what form of packaging would be easier > than just checking out the source from the SCM. > > What do you think would be the right thing to do with it at this > point? Well, there should probably be a tar files with a README. What about adding this to /contrib? -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. +
Bruce Momjian <bruce@momjian.us> wrote: > On Tue, Sep 4, 2012 at 12:11:53PM -0500, Kevin Grittner wrote: >> What do you think would be the right thing to do with it at this >> point? > > Well, there should probably be a tar files with a README. What > about adding this to /contrib? I would be happy to add it to contrib if people want it there. There was some discussion of that when I wrote it, but then there was a feeling that it would be uninteresting once people had streaming replication. I'm skeptical that people will ever lose interest in archiving WAL files, because of the not-uncommon need to keep archival backups or restore to a "point in time" to recover from a mangled table. As long as those needs are there, I suspect that people will want to compress WAL files, including making partially-filled files smaller. -Kevin
On Tue, Sep 4, 2012 at 12:49:40PM -0500, Kevin Grittner wrote: > Bruce Momjian <bruce@momjian.us> wrote: > > On Tue, Sep 4, 2012 at 12:11:53PM -0500, Kevin Grittner wrote: > > >> What do you think would be the right thing to do with it at this > >> point? > > > > Well, there should probably be a tar files with a README. What > > about adding this to /contrib? > > I would be happy to add it to contrib if people want it there. > There was some discussion of that when I wrote it, but then there > was a feeling that it would be uninteresting once people had > streaming replication. I'm skeptical that people will ever lose > interest in archiving WAL files, because of the not-uncommon need to > keep archival backups or restore to a "point in time" to recover > from a mangled table. As long as those needs are there, I suspect > that people will want to compress WAL files, including making > partially-filled files smaller. Agreed. Please move forward on the contrib idea. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. +
On Tue, Sep 4, 2012 at 02:01:54PM -0400, Bruce Momjian wrote: > On Tue, Sep 4, 2012 at 12:49:40PM -0500, Kevin Grittner wrote: > > Bruce Momjian <bruce@momjian.us> wrote: > > > On Tue, Sep 4, 2012 at 12:11:53PM -0500, Kevin Grittner wrote: > > > > >> What do you think would be the right thing to do with it at this > > >> point? > > > > > > Well, there should probably be a tar files with a README. What > > > about adding this to /contrib? > > > > I would be happy to add it to contrib if people want it there. > > There was some discussion of that when I wrote it, but then there > > was a feeling that it would be uninteresting once people had > > streaming replication. I'm skeptical that people will ever lose > > interest in archiving WAL files, because of the not-uncommon need to > > keep archival backups or restore to a "point in time" to recover > > from a mangled table. As long as those needs are there, I suspect > > that people will want to compress WAL files, including making > > partially-filled files smaller. > > Agreed. Please move forward on the contrib idea. Kevin, did this get accomplished? -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. +
Bruce Momjian escribi=F3: >=20 > On Tue, Sep 4, 2012 at 02:01:54PM -0400, Bruce Momjian wrote: > > On Tue, Sep 4, 2012 at 12:49:40PM -0500, Kevin Grittner wrote: > > > Bruce Momjian <bruce@momjian.us> wrote: > > > > On Tue, Sep 4, 2012 at 12:11:53PM -0500, Kevin Grittner wrote: > > > =20 > > > >> What do you think would be the right thing to do with it at this > > > >> point? > > > >=20 > > > > Well, there should probably be a tar files with a README. What > > > > about adding this to /contrib? > > > =20 > > > I would be happy to add it to contrib if people want it there.=20 > > > There was some discussion of that when I wrote it, but then there > > > was a feeling that it would be uninteresting once people had > > > streaming replication. I'm skeptical that people will ever lose > > > interest in archiving WAL files, because of the not-uncommon need t= o > > > keep archival backups or restore to a "point in time" to recover > > > from a mangled table. As long as those needs are there, I suspect > > > that people will want to compress WAL files, including making > > > partially-filled files smaller. > >=20 > > Agreed. Please move forward on the contrib idea. >=20 > Kevin, did this get accomplished? Just to be clear, what we're proposing adding to contrib is clearxlogtail, not pglesslog, right? --=20 =C1lvaro Herrera http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
On Thu, Jan 24, 2013 at 08:43:37PM -0300, Alvaro Herrera wrote: > Bruce Momjian escribió: > > > > On Tue, Sep 4, 2012 at 02:01:54PM -0400, Bruce Momjian wrote: > > > On Tue, Sep 4, 2012 at 12:49:40PM -0500, Kevin Grittner wrote: > > > > Bruce Momjian <bruce@momjian.us> wrote: > > > > > On Tue, Sep 4, 2012 at 12:11:53PM -0500, Kevin Grittner wrote: > > > > > > > > >> What do you think would be the right thing to do with it at this > > > > >> point? > > > > > > > > > > Well, there should probably be a tar files with a README. What > > > > > about adding this to /contrib? > > > > > > > > I would be happy to add it to contrib if people want it there. > > > > There was some discussion of that when I wrote it, but then there > > > > was a feeling that it would be uninteresting once people had > > > > streaming replication. I'm skeptical that people will ever lose > > > > interest in archiving WAL files, because of the not-uncommon need to > > > > keep archival backups or restore to a "point in time" to recover > > > > from a mangled table. As long as those needs are there, I suspect > > > > that people will want to compress WAL files, including making > > > > partially-filled files smaller. > > > > > > Agreed. Please move forward on the contrib idea. > > > > Kevin, did this get accomplished? > > Just to be clear, what we're proposing adding to contrib is > clearxlogtail, not pglesslog, right? I was not clear on exactly what Kevin was going to add. Kevin? -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. +
On 2012-04-09 15:37:09 -0300, Alvaro Herrera wrote: > > Excerpts from Bruce Momjian's message of lun abr 09 15:13:10 -0300 2012: > > On Mon, Apr 09, 2012 at 10:10:34AM -0400, Robert Haas wrote: > > > > This complaint appears to be accurate. I think we should go ahead and > > > remove that mention. > > > > Agreed; removed with the attached patch. I didn't bother keeping the > > gzip mention because I assume there is little value to using without > > pglesslog. > > I'm not sure that assumption holds. I think you're thinking of the hack > that zeroes out the "empty" holes in the middle of data pages; those > didn't compress well unless zeroed out before compression. This tool is > not about that, but rather about removing redundant info from WAL files. > It seems to me that WAL files would be as gzip-compressible regardless > of pglesslog being applied. (Another related tool is clearxlogtail > which zeroes areas from WAL files when they are empty because of an > early switch due to archive timeout). > > The funny thing is, apparently pg_lesslog was intentionally broken by > changing XLR_BKP_REMOVABLE to XLP_BKP_REMOVABLE (different semantics?) > and the fixes to make it compile again look simple. It's a bit strange > that NTT stopped maintaining the tool .. Maybe it wasn't useful anymore? One problem is that it was hard to trust the tool, it had relatively bad bugs several times and it wasn't maintained by pgdg... Andres Freund -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
On Thu, Jan 24, 2013 at 06:45:02PM -0500, Bruce Momjian wrote: > On Thu, Jan 24, 2013 at 08:43:37PM -0300, Alvaro Herrera wrote: > > Bruce Momjian escribió: > > > > > > On Tue, Sep 4, 2012 at 02:01:54PM -0400, Bruce Momjian wrote: > > > > On Tue, Sep 4, 2012 at 12:49:40PM -0500, Kevin Grittner wrote: > > > > > Bruce Momjian <bruce@momjian.us> wrote: > > > > > > On Tue, Sep 4, 2012 at 12:11:53PM -0500, Kevin Grittner wrote: > > > > > > > > > > >> What do you think would be the right thing to do with it at this > > > > > >> point? > > > > > > > > > > > > Well, there should probably be a tar files with a README. What > > > > > > about adding this to /contrib? > > > > > > > > > > I would be happy to add it to contrib if people want it there. > > > > > There was some discussion of that when I wrote it, but then there > > > > > was a feeling that it would be uninteresting once people had > > > > > streaming replication. I'm skeptical that people will ever lose > > > > > interest in archiving WAL files, because of the not-uncommon need to > > > > > keep archival backups or restore to a "point in time" to recover > > > > > from a mangled table. As long as those needs are there, I suspect > > > > > that people will want to compress WAL files, including making > > > > > partially-filled files smaller. > > > > > > > > Agreed. Please move forward on the contrib idea. > > > > > > Kevin, did this get accomplished? > > > > Just to be clear, what we're proposing adding to contrib is > > clearxlogtail, not pglesslog, right? > > I was not clear on exactly what Kevin was going to add. Kevin? Any news on this? -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. +