Thread: [v9.2] Start new timeline for PITR
Folks, The nice people at VMware, where I work, have come up with a small patch to allow PITR to create a new timeline. This is useful in cases where you're using filesystem snapshots of $PGDATA which may be old. PFA a patch implementing and documenting same :) Cheers, David. -- David Fetter <david@fetter.org> http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fetter@gmail.com iCal: webcal://www.tripit.com/feed/ical/people/david74/tripit.ics Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate
Attachment
On 6/9/11 4:51 PM, David Fetter wrote: > Folks, > > The nice people at VMware, where I work, have come up with a small > patch to allow PITR to create a new timeline. This is useful in cases > where you're using filesystem snapshots of $PGDATA which may be old. > > PFA a patch implementing and documenting same :) Can you explain here in email what the specific goals and expected behavior of the option are? -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com
David Fetter <david@fetter.org> writes:
> The nice people at VMware, where I work, have come up with a small
> patch to allow PITR to create a new timeline. This is useful in cases
> where you're using filesystem snapshots of $PGDATA which may be old.
Huh? We already start a new timeline when doing a non-crash-recovery
replay scenario.
The code looks pretty confused too, which makes it difficult to
reverse-engineer what your point is.
regards, tom lane
On Thu, Jun 9, 2011 at 8:59 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > David Fetter <david@fetter.org> writes: >> The nice people at VMware, where I work, have come up with a small >> patch to allow PITR to create a new timeline. This is useful in cases >> where you're using filesystem snapshots of $PGDATA which may be old. > > Huh? We already start a new timeline when doing a non-crash-recovery > replay scenario. > > The code looks pretty confused too, which makes it difficult to > reverse-engineer what your point is. I am guessing that they are taking a filesystem snapshot, and then using that to fire up PG. So to PG it looks like a crash recovery, but they want a new timeline anyway. <waves hands> -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
On Fri, Jun 10, 2011 at 01:20:25AM -0400, Robert Haas wrote: > On Thu, Jun 9, 2011 at 8:59 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > > David Fetter <david@fetter.org> writes: > >> The nice people at VMware, where I work, have come up with a small > >> patch to allow PITR to create a new timeline. This is useful in cases > >> where you're using filesystem snapshots of $PGDATA which may be old. > > > > Huh? We already start a new timeline when doing a non-crash-recovery > > replay scenario. > > > > The code looks pretty confused too, which makes it difficult to > > reverse-engineer what your point is. > > I am guessing that they are taking a filesystem snapshot, and then > using that to fire up PG. So to PG it looks like a crash recovery, > but they want a new timeline anyway. > > <waves hands> That's pretty much it. More detail: Let's imagine we're taking filesystem snapshots each day by whatever means. We're also archiving xlogs, but only have space for 48 hours' worth. Now we want to recover to 3 days ago, but there are no WALs from that time, so we do a crash recovery from the filesystem snapshot. Doing continuous archiving from this conflicts with the existing WALs, which we solve by creating a new timeline. This also allows subsequent PITR to other times on the original timeline. Josh B pointed out that since this option to true conflicts with another option, having both should prevent recovery from even starting, and I'll work up a patch for this tonight or at latest tomorrow. Cheers, David. -- David Fetter <david@fetter.org> http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fetter@gmail.com iCal: webcal://www.tripit.com/feed/ical/people/david74/tripit.ics Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate
David, > Let's imagine we're taking filesystem snapshots each day by whatever > means. We're also archiving xlogs, but only have space for 48 hours' > worth. Now we want to recover to 3 days ago, but there are no WALs > from that time, so we do a crash recovery from the filesystem > snapshot. Doing continuous archiving from this conflicts with the > existing WALs, which we solve by creating a new timeline. How is this different from just changing the recovery_command? -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com
David, > Let's imagine we're taking filesystem snapshots each day by whatever > means. We're also archiving xlogs, but only have space for 48 hours' > worth. Now we want to recover to 3 days ago, but there are no WALs > from that time, so we do a crash recovery from the filesystem > snapshot. Doing continuous archiving from this conflicts with the > existing WALs, which we solve by creating a new timeline. How is this different from just changing the recovery_command? -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com
On Fri, Jun 10, 2011 at 2:53 PM, Josh Berkus <josh@agliodbs.com> wrote: >> Let's imagine we're taking filesystem snapshots each day by whatever >> means. We're also archiving xlogs, but only have space for 48 hours' >> worth. Now we want to recover to 3 days ago, but there are no WALs >> from that time, so we do a crash recovery from the filesystem >> snapshot. Doing continuous archiving from this conflicts with the >> existing WALs, which we solve by creating a new timeline. > > How is this different from just changing the recovery_command? *scratches head* How is it the same? -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
On 10.06.2011 22:34, Robert Haas wrote: > On Fri, Jun 10, 2011 at 2:53 PM, Josh Berkus<josh@agliodbs.com> wrote: >>> Let's imagine we're taking filesystem snapshots each day by whatever >>> means. We're also archiving xlogs, but only have space for 48 hours' >>> worth. Now we want to recover to 3 days ago, but there are no WALs >>> from that time, so we do a crash recovery from the filesystem >>> snapshot. Doing continuous archiving from this conflicts with the >>> existing WALs, which we solve by creating a new timeline. >> >> How is this different from just changing the recovery_command? > > *scratches head* > > How is it the same? Creating a dummy recovery.conf with bogus recovery_command would do the trick. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com
On 6/10/11 12:34 PM, Robert Haas wrote: > On Fri, Jun 10, 2011 at 2:53 PM, Josh Berkus <josh@agliodbs.com> wrote: >>> Let's imagine we're taking filesystem snapshots each day by whatever >>> means. We're also archiving xlogs, but only have space for 48 hours' >>> worth. Now we want to recover to 3 days ago, but there are no WALs >>> from that time, so we do a crash recovery from the filesystem >>> snapshot. Doing continuous archiving from this conflicts with the >>> existing WALs, which we solve by creating a new timeline. >> >> How is this different from just changing the recovery_command? > > *scratches head* > > How is it the same? Well, presumably I can just change recovery_command to recover from an empty directory. Then the PITR copy will just come up as soon as it finishes processing local snapshot WAL, and it'll start its own timeline. How is this different from what the patch does? -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com
On Fri, Jun 10, 2011 at 11:30 AM, David Fetter <david@fetter.org> wrote: > > This also allows subsequent PITR to other times on the original > timeline. > > Josh B pointed out that since this option to true conflicts with > another option, having both should prevent recovery from even > starting, and I'll work up a patch for this tonight or at latest > tomorrow. > Hi, Are you still working on this? should we expect a new patch? -- Jaime Casanova www.2ndQuadrant.com Professional PostgreSQL: Soporte 24x7 y capacitación
On Fri, Jun 17, 2011 at 09:57:13AM -0500, Jaime Casanova wrote: > On Fri, Jun 10, 2011 at 11:30 AM, David Fetter <david@fetter.org> wrote: > > > > This also allows subsequent PITR to other times on the original > > timeline. > > > > Josh B pointed out that since this option to true conflicts with > > another option, having both should prevent recovery from even > > starting, and I'll work up a patch for this tonight or at latest > > tomorrow. > > Hi, > > Are you still working on this? should we expect a new patch? Yes, sorry about that. I let work get on top of me. Will try for a new patch this evening. Cheers, David. -- David Fetter <david@fetter.org> http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fetter@gmail.com iCal: webcal://www.tripit.com/feed/ical/people/david74/tripit.ics Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate
On Fri, Jun 17, 2011 at 1:54 PM, David Fetter <david@fetter.org> wrote: > On Fri, Jun 17, 2011 at 09:57:13AM -0500, Jaime Casanova wrote: >> >> Are you still working on this? should we expect a new patch? > > Yes, sorry about that. I let work get on top of me. Will try for a > new patch this evening. > ok... i will wait it to review... just in advance, i really don't like this name "create_new_timeline"... it will drive confusion -- Jaime Casanova www.2ndQuadrant.com Professional PostgreSQL: Soporte 24x7 y capacitación