Thread: Bug-fix and new feature of pg_lesslog is released
It is an announcement that long-waited bug-fix of pg_lesslog is now released. This includes the following. 1. Error in calculation of GiST-related WAL record length was fixed. 2. pg_compresslog now has an option to print WAL segment analysis. 3. New test script is added which analyzes what WAL record was generated and if all the recovery was done successfully. 2 and 3 are added to ensure the tools run correctly. Because 8.4 has additional WAL records, I made release material separate for each PG version. You may visit the project page http://pgfoundry.org/frs/?group_id=1000310 to download materials. I acknowledge PG hackers who gave valuable comments/advices. Especially Oleg Bartunov, Teodor Sigaev and Alvaro Herrera gave very helpful information how to generate specific type of WAL records. As Bruce suggested, this can be included in PostgreSQL test suits and could be included in contrib module if it is ported to PostgreSQL 9.1. Best Regards; ---------- Koichi Suzuki # pg_lesslog is a project to compress large WAL segments when they're stored and restore then for recovery. ---------------------------(end of broadcast)--------------------------- -To unsubscribe from this list, send an email to: pgsql-announce-unsubscribe@postgresql.org
Koichi Suzuki wrote: > It is an announcement that long-waited bug-fix of pg_lesslog is now > released. This includes the following. > > 1. Error in calculation of GiST-related WAL record length was fixed. > 2. pg_compresslog now has an option to print WAL segment analysis. > 3. New test script is added which analyzes what WAL record was > generated and if all the recovery was done successfully. > > 2 and 3 are added to ensure the tools run correctly. > > Because 8.4 has additional WAL records, I made release material > separate for each PG version. You may visit the project page > http://pgfoundry.org/frs/?group_id=1000310 > to download materials. > > I acknowledge PG hackers who gave valuable comments/advices. > Especially Oleg Bartunov, Teodor Sigaev and Alvaro Herrera gave very > helpful information how to generate specific type of WAL records. > > As Bruce suggested, this can be included in PostgreSQL test suits and > could be included in contrib module if it is ported to PostgreSQL 9.1. [ email to hackers list only ] Yes, I would love to get this into /contrib for PG 9.1! -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com
Bruce Momjian <bruce@momjian.us> writes:
> Yes, I would love to get this into /contrib for PG 9.1!
How much are people really going to care about pg_lesslog now that
we've got streaming replication? There might be some small use-case
still left, but it's hard to believe that it would be worth carrying
it in contrib.
regards, tom lane
On Wed, May 12, 2010 at 11:03 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > Bruce Momjian <bruce@momjian.us> writes: >> Yes, I would love to get this into /contrib for PG 9.1! > > How much are people really going to care about pg_lesslog now that > we've got streaming replication? There might be some small use-case > still left, but it's hard to believe that it would be worth carrying > it in contrib. Until we have a tool to use streaming replication to actually archive backups, there's still a fairly large usecase for it... -- Magnus HaganderMe: http://www.hagander.net/Work: http://www.redpill-linpro.com/
On Wed, 2010-05-12 at 23:09 +0200, Magnus Hagander wrote: > On Wed, May 12, 2010 at 11:03 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > > Bruce Momjian <bruce@momjian.us> writes: > >> Yes, I would love to get this into /contrib for PG 9.1! > > > > How much are people really going to care about pg_lesslog now that > > we've got streaming replication? There might be some small use-case > > still left, but it's hard to believe that it would be worth carrying > > it in contrib. > > Until we have a tool to use streaming replication to actually archive > backups, there's still a fairly large usecase for it... Not to mention, WAN based standby, timed standby etc... Joshua D. Drake > > > -- > Magnus Hagander > Me: http://www.hagander.net/ > Work: http://www.redpill-linpro.com/ > -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564 Consulting, Training, Support, Custom Development, Engineering
> Until we have a tool to use streaming replication to actually archive
> backups, there's still a fairly large usecase for it...
yes, pglesslog is for PITR. We have a client who keeps 3 days of log
files a time for forensic reasons, and I doubt they're the only one.
-- -- Josh Berkus PostgreSQL Experts Inc.
http://www.pgexperts.com
On Wed, 2010-05-12 at 23:09 +0200, Magnus Hagander wrote: > On Wed, May 12, 2010 at 11:03 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > > Bruce Momjian <bruce@momjian.us> writes: > >> Yes, I would love to get this into /contrib for PG 9.1! > > > > How much are people really going to care about pg_lesslog now that > > we've got streaming replication? There might be some small use-case > > still left, but it's hard to believe that it would be worth carrying > > it in contrib. > > Until we have a tool to use streaming replication to actually archive > backups, there's still a fairly large usecase for it... Not to mention, WAN based standby, timed standby etc... Joshua D. Drake > > > -- > Magnus Hagander > Me: http://www.hagander.net/ > Work: http://www.redpill-linpro.com/ > -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564 Consulting, Training, Support, Custom Development, Engineering
Tom Lane <tgl@sss.pgh.pa.us> wrote: > Bruce Momjian <bruce@momjian.us> writes: > > Yes, I would love to get this into /contrib for PG 9.1! > > How much are people really going to care about pg_lesslog now that > we've got streaming replication? There might be some small use-case > still left, but it's hard to believe that it would be worth carrying > it in contrib. I hope pg_lesslog would work as a WAL filter of streaming replication. It might be hard-coded in WAL sender, or be an addon based on a new common filtering infrastructure of WAL streaming. Also, there is a long-standing issue in pg_lesslog; It slows down recovery because we need to read data pages before write in recovery. We're avoiding reading pages for full-page image in 8.3, but pg_lesslog will disable the optimization. Recovery routine in core also needs to be adjusted to use read-ahead, like posix_fadvise(). There was another idea, full-page image logs separated with WAL logging. In theory, full-page images don't have to be written at commit, but only by writing corresponding data pages, I'm not sure whether it is an actually good idea or not, but if we go the direction, we won't need pg_lesslog. Regards, --- Takahiro Itagaki NTT Open Source Software Center
Thanks a lot for the comment/advice. Yes, full page backup block considerablly shortens the recovery time. As we discussed about two years ago, I have a solution accelerate the recovery even without full page image. I'd like to submit this solution to the community again. When I evaluated this two years ago, recovery speedwas as good as those with full page image, depending upon application and tuning, of course. This is a separate tool and can be used in various scenes. Regards; ---------- Koichi Suzuki 2010/5/13 Takahiro Itagaki <itagaki.takahiro@oss.ntt.co.jp>: > > Tom Lane <tgl@sss.pgh.pa.us> wrote: > >> Bruce Momjian <bruce@momjian.us> writes: >> > Yes, I would love to get this into /contrib for PG 9.1! >> >> How much are people really going to care about pg_lesslog now that >> we've got streaming replication? There might be some small use-case >> still left, but it's hard to believe that it would be worth carrying >> it in contrib. > > I hope pg_lesslog would work as a WAL filter of streaming replication. > It might be hard-coded in WAL sender, or be an addon based on a new > common filtering infrastructure of WAL streaming. > > Also, there is a long-standing issue in pg_lesslog; It slows down recovery > because we need to read data pages before write in recovery. We're avoiding > reading pages for full-page image in 8.3, but pg_lesslog will disable > the optimization. Recovery routine in core also needs to be adjusted to use > read-ahead, like posix_fadvise(). > > There was another idea, full-page image logs separated with WAL logging. > In theory, full-page images don't have to be written at commit, but only > by writing corresponding data pages, I'm not sure whether it is an actually > good idea or not, but if we go the direction, we won't need pg_lesslog. > > Regards, > --- > Takahiro Itagaki > NTT Open Source Software Center > > >
WAL streming filter is an interesting idea. We can leave full page backup for local recovery and decrease the amount of WAL to transfer. If SR writes WAL in record by record basis, implementation will be simple. If SR writes WAL in block by block basis, WAL records may be divided from transfer to transfer and it is not simple to handle. Regards; ---------- Koichi Suzuki 2010/5/13 Takahiro Itagaki <itagaki.takahiro@oss.ntt.co.jp>: > > Tom Lane <tgl@sss.pgh.pa.us> wrote: > >> Bruce Momjian <bruce@momjian.us> writes: >> > Yes, I would love to get this into /contrib for PG 9.1! >> >> How much are people really going to care about pg_lesslog now that >> we've got streaming replication? There might be some small use-case >> still left, but it's hard to believe that it would be worth carrying >> it in contrib. > > I hope pg_lesslog would work as a WAL filter of streaming replication. > It might be hard-coded in WAL sender, or be an addon based on a new > common filtering infrastructure of WAL streaming. > > Also, there is a long-standing issue in pg_lesslog; It slows down recovery > because we need to read data pages before write in recovery. We're avoiding > reading pages for full-page image in 8.3, but pg_lesslog will disable > the optimization. Recovery routine in core also needs to be adjusted to use > read-ahead, like posix_fadvise(). > > There was another idea, full-page image logs separated with WAL logging. > In theory, full-page images don't have to be written at commit, but only > by writing corresponding data pages, I'm not sure whether it is an actually > good idea or not, but if we go the direction, we won't need pg_lesslog. > > Regards, > --- > Takahiro Itagaki > NTT Open Source Software Center > > >