Thread: Point in time recovery 20020822_01_pitr.patch

Point in time recovery 20020822_01_pitr.patch

From
"J. R. Nield"
Date:
Here's a patch to do point-in-time recovery against 7.3 head. Still to
be resolved in this patch:

    * CREATE/DROP database are still unlogged. dbcommands.c needs a
re-write to make that possible, since all the file and directory
operations will need to be logged, as will the data.

    * Is it really safe to do block-aligned reads to copy the database
files once (if) we remove backup blocks from the log? Should support be
added to go through the buffer manager?


New Files:
    ./doc/FAQ_PITR
    ./src/backend/commands/recoverycmds.c
    ./src/include/commands/recoverycmds.h


--
J. R. Nield
jrnield@usol.com



Attachment

Re: Point in time recovery 20020822_01_pitr.patch

From
Bruce Momjian
Date:
Patch applied.  Thanks.




Ha.  Ha.  Just messing with you guys.  I didn't apply anything, but this
does give us something to talk about.  Thanks.

---------------------------------------------------------------------------


J. R. Nield wrote:
> Here's a patch to do point-in-time recovery against 7.3 head. Still to
> be resolved in this patch:
>
>     * CREATE/DROP database are still unlogged. dbcommands.c needs a
> re-write to make that possible, since all the file and directory
> operations will need to be logged, as will the data.
>
>     * Is it really safe to do block-aligned reads to copy the database
> files once (if) we remove backup blocks from the log? Should support be
> added to go through the buffer manager?
>
>
> New Files:
>     ./doc/FAQ_PITR
>     ./src/backend/commands/recoverycmds.c
>     ./src/include/commands/recoverycmds.h
>
>
> --
> J. R. Nield
> jrnield@usol.com
>
>

[ Attachment, skipping... ]

--
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Re: Point in time recovery 20020822_01_pitr.patch

From
Tom Lane
Date:
"J. R. Nield" <jrnield@usol.com> writes:
>     * CREATE/DROP database are still unlogged. dbcommands.c needs a
> re-write to make that possible, since all the file and directory
> operations will need to be logged, as will the data.

Why don't you log the operations symbolically, viz "create database foo
with template bar"?  I can't see any reason to insist on a finer level
of detail than that.


Just so you know: my feeling is this is already too late for 7.3.

            regards, tom lane

Re: Point in time recovery 20020822_01_pitr.patch

From
Bruce Momjian
Date:
J. R. Nield wrote:
> Here's a patch to do point-in-time recovery against 7.3 head. Still to
> be resolved in this patch:
>
>     * CREATE/DROP database are still unlogged. dbcommands.c needs a
> re-write to make that possible, since all the file and directory
> operations will need to be logged, as will the data.
>
>     * Is it really safe to do block-aligned reads to copy the database
> files once (if) we remove backup blocks from the log? Should support be
> added to go through the buffer manager?
>
>
> New Files:
>     ./doc/FAQ_PITR
>     ./src/backend/commands/recoverycmds.c
>     ./src/include/commands/recoverycmds.h

Here is quick overview of the patch.  First, it starts with
documentation about how the user interacts with the PITR system.  Then
it goes into a large chunk of code that logs and recovers from a variety
of database changes that we don't currently log because they aren't
needed for crash recovery but are required for PITR recovery.  Next is a
section that does the actual file backups for PITR, and finally, there
is grammar code to support the new PITR commands.

--
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Re: Point in time recovery 20020822_01_pitr.patch

From
Bruce Momjian
Date:
Tom Lane wrote:
> "J. R. Nield" <jrnield@usol.com> writes:
> >     * CREATE/DROP database are still unlogged. dbcommands.c needs a
> > re-write to make that possible, since all the file and directory
> > operations will need to be logged, as will the data.
>
> Why don't you log the operations symbolically, viz "create database foo
> with template bar"?  I can't see any reason to insist on a finer level
> of detail than that.
>
>
> Just so you know: my feeling is this is already too late for 7.3.

I understand Tom's concerns.  We have already talked about this on the
phone because we anticipated this may happen.

We are one week from beta.   I think what we should do is to give
everyone a day to look over the patch and submit their ideas about it to
the group.  We can then review the possible scenarios we can take:

    apply and contine to September 1 beta
    apply and delay beta until October 1
    create a patch file against 7.3 for distribution
    wait for 7.4 and do not distribute

--
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Re: Point in time recovery 20020822_01_pitr.patch

From
"Christopher Kings-Lynne"
Date:
> I understand Tom's concerns.  We have already talked about this on the
> phone because we anticipated this may happen.
>
> We are one week from beta.   I think what we should do is to give
> everyone a day to look over the patch and submit their ideas about it to
> the group.  We can then review the possible scenarios we can take:
>
>     apply and contine to September 1 beta
>     apply and delay beta until October 1
>     create a patch file against 7.3 for distribution
>     wait for 7.4 and do not distribute

I too understand Tom's concerns.  It is a lot of changes in various
important parts of code and it won't have been tested enough, etc., etc.  I
say leave it for 7.4.  Distributing a patch against 7.3 will be very
difficult to keep up-to-date as the CVS tree changes.  Maybe we could look
to aim to have win32 native and PITR for 7.4 and so as soon as they're in,
working and tested we just release 7.4 - why not?  I guess maybe the wire
protocol might change as well.

And what about neil's PREPARE?

Chris


Re: Point in time recovery 20020822_01_pitr.patch

From
Tom Lane
Date:
"Christopher Kings-Lynne" <chriskl@familyhealth.com.au> writes:
> And what about neil's PREPARE?

PREPARE I think we will have reviewed and in there for beta; I just need
to find a couple of hours for it.  (I already looked at it once but
want to take another look at the revised patch.)

PREPARE is hardly of the magnitude of PITR anyway: the worst-case
scenario with PREPARE is that it's broken and people can't safely
use it in 7.3.  It can't possibly hurt you if you don't use it.
In contrast, the worst-case scenario with PITR is, um, dire.

            regards, tom lane

Re: Point in time recovery 20020822_01_pitr.patch

From
"Marc G. Fournier"
Date:
the fact that JR acknowledges issues that are yet to be resolved, and that
we are in the final week before beta ... I agree with Tom that this should
not go into v7.3, and be left for v7.4 ...

On Fri, 23 Aug 2002, Bruce Momjian wrote:

> J. R. Nield wrote:
> > Here's a patch to do point-in-time recovery against 7.3 head. Still to
> > be resolved in this patch:
> >
> >     * CREATE/DROP database are still unlogged. dbcommands.c needs a
> > re-write to make that possible, since all the file and directory
> > operations will need to be logged, as will the data.
> >
> >     * Is it really safe to do block-aligned reads to copy the database
> > files once (if) we remove backup blocks from the log? Should support be
> > added to go through the buffer manager?
> >
> >
> > New Files:
> >     ./doc/FAQ_PITR
> >     ./src/backend/commands/recoverycmds.c
> >     ./src/include/commands/recoverycmds.h
>
> Here is quick overview of the patch.  First, it starts with
> documentation about how the user interacts with the PITR system.  Then
> it goes into a large chunk of code that logs and recovers from a variety
> of database changes that we don't currently log because they aren't
> needed for crash recovery but are required for PITR recovery.  Next is a
> section that does the actual file backups for PITR, and finally, there
> is grammar code to support the new PITR commands.
>
> --
>   Bruce Momjian                        |  http://candle.pha.pa.us
>   pgman@candle.pha.pa.us               |  (610) 359-1001
>   +  If your life is a hard drive,     |  13 Roberts Road
>   +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073
>
> ---------------------------(end of broadcast)---------------------------
> TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org
>


Re: Point in time recovery 20020822_01_pitr.patch

From
"J. R. Nield"
Date:
On Fri, 2002-08-23 at 00:29, Tom Lane wrote:
> "J. R. Nield" <jrnield@usol.com> writes:
> >     * CREATE/DROP database are still unlogged. dbcommands.c needs a
> > re-write to make that possible, since all the file and directory
> > operations will need to be logged, as will the data.
>
> Why don't you log the operations symbolically, viz "create database foo
> with template bar"?  I can't see any reason to insist on a finer level
> of detail than that.

If you do symbolic logging like that, it forecloses any chance of adding
individual relation recovery, because the template might be ahead of the
log. That's main reason I avoided doing this with index builds. I don't
know if we will want that feature, but I didn't want to prevent us from
adding it.

>
>
> Just so you know: my feeling is this is already too late for 7.3.
>

I'm not in a position to answer this. I'll do whatever I can to address
concerns as they arise, and I'll continue to review my own work. Clearly
this is not a good area to find bugs "in the field".

>             regards, tom lane
>
--
J. R. Nield
jrnield@usol.com




Re: Point in time recovery 20020822_01_pitr.patch

From
Tom Lane
Date:
"J. R. Nield" <jrnield@usol.com> writes:
> On Fri, 2002-08-23 at 00:29, Tom Lane wrote:
>> Why don't you log the operations symbolically, viz "create database foo
>> with template bar"?  I can't see any reason to insist on a finer level
>> of detail than that.

> If you do symbolic logging like that, it forecloses any chance of adding
> individual relation recovery, because the template might be ahead of the
> log.

We are already assuming that the template database is stable while it's
being copied.  While there are obvious risks in that assumption, I don't
think that you need to eliminate the assumption as an essential
component of PITR ... and I think that there are a number of issues
that'd have to be solved that are completely unrelated to PITR (eg, how
do you lock a table that's in a database you aren't in, and indeed that
you don't even know exists, because you can't read the pg_class table
that describes it?)  My advice: this is a job not to tackle for version
1, and maybe not for version 2, 3, or 4...

            regards, tom lane

Re: prepare patch

From
Barry Lind
Date:
The sooner the better on this.  I really want to get this functionality
into the jdbc driver in 7.3.  It is going to be a big performance win
for my applications.

thanks,
--Barry

Tom Lane wrote:

 >"Christopher Kings-Lynne" <chriskl@familyhealth.com.au> writes:
 >
 >
 >>And what about neil's PREPARE?
 >>
 >>
 >
 >PREPARE I think we will have reviewed and in there for beta; I just need
 >to find a couple of hours for it.  (I already looked at it once but
 >want to take another look at the revised patch.)
 >
 >PREPARE is hardly of the magnitude of PITR anyway: the worst-case
 >scenario with PREPARE is that it's broken and people can't safely
 >use it in 7.3.  It can't possibly hurt you if you don't use it.
 >In contrast, the worst-case scenario with PITR is, um, dire.
 >
 >
        regards, tom lane
 >
 >---------------------------(end of broadcast)---------------------------
 >TIP 3: if posting/reading through Usenet, please send an appropriate
 >subscribe-nomail command to majordomo@postgresql.org so that your
 >message can get through to the mailing list cleanly
 >
 >
 >