Thread: PITR Phase 1 - Full set of patches

PITR Phase 1 - Full set of patches

From
Simon Riggs
Date:
Full set of patches, including both earlier reported bugs fixed.

This note was originally posted at Tue, 27 Apr 2004 23:30:39 +0100, but
for some reason hasn't yet appeared on the [HACKERS] list, so I am now
reposting this. Trying PATCHES list now...

Full usage instructions (for TESTING only!)

Patches:
guc.c.patch goes to src/backend/utils/misc/guc.c
xlog.c.patch goes to src/backend/access/transam/xlog.c
xlog.h.patch goes to src/include/access/xlog.h
Makefile.patch goes to src/bin/Makefile
pgarch.tar adds directory and files for src/bin/pg_arch

Build:
pg_arch builds as part of the full source tree...

General Execution
1. create pg_rlog directory beneath PGDATA
2. add (optional) line to postgresql.conf
wal_debug=1
3. add (required) line to postgresql.conf
wal_archive=true
4. execute pg_arch /your-archive-dir $PGDATA
5. startup new postmaster
6. execute workload of choice to force the creation of xlogs
7. watch the execution...

Recovery Test:
1. Work out ways of testing the logical validity of data
2. While postmaster is UP: take a full physical backup of PGDATA
3. wait for some time while transaction archives get posted
4. perform your choice of destructive action on postgresql
5. mv backup copy to a restore location, possibly where it just was
6. mv ALL transaction logs taken SINCE full backup
7. startup postmaster and watch
8. execute tests from 1, to ensure recovery successful

...my patch building experience is less than some might
expect so there are various possible annoyances here. I am on hand to
help and to learn by my mistakes.

Report bugs to me at simon@2ndquadrant.com and/or to the [HACKERS] list
pgsql-hackers@postgresql.org

Thanks,

Best Regards

Simon Riggs
2nd Quadrant
http://www.2ndquadrant.com


Attachment

Re: PITR Phase 1 - Full set of patches

From
Alvaro Herrera
Date:
On Thu, Apr 29, 2004 at 11:21:53PM +0100, Simon Riggs wrote:

> ...my patch building experience is less than some might
> expect so there are various possible annoyances here. I am on hand to
> help and to learn by my mistakes.

I see you attached one patch for every file ... this works, but is error
prone.  May I suggest another approach:

1. get CVSup if you can.  I have Linux, so I got the ezm3 package and
the CVSup package.  Compile/install ezm3 (it's a Modula-3 compiler)
first and then CVSup.  Google for them, I don't have URLs.

2. get the whole Postgres repository using CVSup with this config file:
-------->-------->-------->-------->-------->-------->-------->-------->
*default host=cvsup.postgresql.org
*default compress
*default release=cvs
*default delete use-rel-suffix
# *default tag=.
# base directory where CVSup will store its 'bookmarks' file(s)
*default base=/home/alvherre/cvs

# prefix directory where CVSup will store the actual distribution(s)
*default prefix=/home/alvherre/cvs

# complete distribution
repository
-------->-------->-------->-------->-------->-------->-------->-------->

This will create a "cvsroot".  Adjust the path, obviously.  Note that
case matters because I have a "cvs" directory with the cvsroot, and a
"CVS" directory for the checkouts.  This may be a stupid idea but it is
how I did it some time ago :-)

3. Checkout a couple of trees
cvs -d /home/alvherre/cvs co pgsql

I have several pgsql trees:

/home/alvherre/CVS/pgsql/source/00orig

this is the pristine CVS tip.

My first modification in
/home/alvherre/CVS/pgsql/source/01xact

second in
/home/alvherre/CVS/pgsql/source/02blahblah
etc, you get the idea.  (I have a "74_rel" and "73_rel" too, just in
case.)

Then I created a "build" tree, in /home/alvherre/CVS/pgsql/build.  So
each time I want to compile, I create a
/home/alvherre/CVS/pgsql/build/00orig, cd to it, and then do
../../source/00orig/configure --enable-debug [etc] --prefix=/home/alvherre/CVS/pgsql/install/00orig


4. To generate a patch, just do
cd /home/alvherre/CVS/pgsql/01xact
cvs -q diff

Note that sometimes I do things incrementally, so I do "incremental
diffs" by commands like
cd /home/alvherre/CVS/pgsql/source
diff -cr --exclude-from=ignore-diff 01xact 02blahblah

The ignore-diff file is there because some files are built on the source
tree and generate a lot of irrelevant differences.  It contains

-------->-------->-------->-------->-------->-------->-------->-------->
pl_gram.c
pl.tab.h
pl_scan.c
psqlscan.c
sql_help.h
preproc.c
preproc.h
pgc.c
guc-file.c
fmgrtab.c
fmgroids.h
parse.h
gram.c
scan.c
bootparse.c
bootstrap_tokens.h
bootscanner.c
*~
tags
CVS
.*sw[poq]
-------->-------->-------->-------->-------->-------->-------->-------->

So, by this time you are wondering why do I use a number as the start of
the tree name.  Well, this is because I have a script which allows me to
- build (configure, make)
- install (make install)
- initdb
- run a server
- run a psql client
- run a "make check" against a running server

And I may want to do any combination of the above with any combination
of the pgsql trees I have, simultaneously.  So I picked a random number
as "base", and the scripts takes the sum of this number and the number
at the start of tree name, and uses it as the port (--with-pgport).  So
I can run all servers and clients, with no conflict.

If there is any interest I can post the script too.  It's somewhat ugly
but it has worked for me.

--
Alvaro Herrera (<alvherre[a]dcc.uchile.cl>)
"La virtud es el justo medio entre dos defectos" (Aristóteles)

Re: PITR Phase 1 - Full set of patches

From
Alvaro Herrera
Date:
On Thu, Apr 29, 2004 at 09:58:49PM -0400, Alvaro Herrera wrote:

> Note that sometimes I do things incrementally, so I do "incremental
> diffs" by commands like
> cd /home/alvherre/CVS/pgsql/source
> diff -cr --exclude-from=ignore-diff 01xact 02blahblah
       ^^^

Note that this should be -Ncr -- I don't create a lot of new files ...

--
Alvaro Herrera (<alvherre[a]dcc.uchile.cl>)
"Cuando no hay humildad las personas se degradan" (A. Christie)

Re: PITR Phase 1 - Full set of patches

From
Simon Riggs
Date:
On Fri, 2004-04-30 at 03:12, Alvaro Herrera wrote:
> On Thu, Apr 29, 2004 at 09:58:49PM -0400, Alvaro Herrera wrote:
>
> > Note that sometimes I do things incrementally, so I do "incremental
> > diffs" by commands like
> > cd /home/alvherre/CVS/pgsql/source
> > diff -cr --exclude-from=ignore-diff 01xact 02blahblah
>        ^^^
>
> Note that this should be -Ncr -- I don't create a lot of new files ...

Thanks very much for the detailed help. I don't create a lot of new
files either...

Best Regards, Simon Riggs