Thread: Postgres-R source code release

Postgres-R source code release

From
Markus Wanner
Date:
Dear Hackers,

it has been two years, since I've presented my work on Postgres-R for
the first time to a broader audience in Toronto. I continued maintaining
that code in my spare time, but to be honest, I didn't have much time
for it.

As much as I'd like to change that, I now think the best thing for the
project itself is to open it up and release the source code. I'd very
much like to get others aboard and turn Postgres-R into a real community
project again. Of course, the recent core statement about wanting an
integrated replication solution also influenced my decision, even though
I realize that Postgres-R is way more ambitious path than a log shipping
solution.

As a first step towards a community project, I've cleaned up the code
and tried to comment and document it as good as I can. You are welcome
to download the latest patch from http://www.postgres-r.org/downloads/.
It expects to be applied against today's CVS HEAD. You'll find a README
and a pretty exhaustive TODO file under src/backend/replication/.

I plan to write a series of more technical mails following this one, to
start the discussion about the internals of Postgres-R and decisions I
made. That should give some starting points for those interested it the
project.

In case you are wondering: I've got married and I'm glad to tell you
that I changed my very German sounding name Schiltknecht to a more
pronounceable one: my new surname is Wanner.

Looking forward to inspiring discussions with fellow hackers.

Regards

Markus Wanner

 configure.in                                  |    9
 src/Makefile.global.in                        |    1
 src/backend/Makefile                          |    4
 src/backend/access/transam/varsup.c           |   23
 src/backend/access/transam/xact.c             |  115 +
 src/backend/bootstrap/bootstrap.c             |    3
 src/backend/catalog/Makefile                  |    4
 src/backend/catalog/catalog.c                 |    8
 src/backend/catalog/index.c                   |   10
 src/backend/commands/Makefile                 |    4
 src/backend/commands/aggregatecmds.c          |   25
 src/backend/commands/comment.c                |   25
 src/backend/commands/conversioncmds.c         |   25
 src/backend/commands/dbcommands.c             |  220 ++-
 src/backend/commands/functioncmds.c           |   64
 src/backend/commands/indexcmds.c              |    9
 src/backend/commands/lockcmds.c               |   10
 src/backend/commands/opclasscmds.c            |   75 +
 src/backend/commands/operatorcmds.c           |   24
 src/backend/commands/prepare.c                |   20
 src/backend/commands/proclang.c               |   32
 src/backend/commands/replicacmds.c            |  188 ++
 src/backend/commands/schemacmds.c             |   29
 src/backend/commands/sequence.c               |  193 ++
 src/backend/commands/tablecmds.c              |   44
 src/backend/commands/trigger.c                |   30
 src/backend/commands/tsearchcmds.c            |  136 +
 src/backend/commands/typecmds.c               |   69
 src/backend/commands/variable.c               |   68
 src/backend/executor/execMain.c               |  651 +++++++++
 src/backend/executor/execUtils.c              |    9
 src/backend/nodes/copyfuncs.c                 |   14
 src/backend/nodes/equalfuncs.c                |   12
 src/backend/parser/gram.y                     |   92 +
 src/backend/parser/keywords.c                 |    6
 src/backend/postmaster/postmaster.c           |  349 ++++
 src/backend/replication/HISTORY               |   33
 src/backend/replication/Makefile              |   17
 src/backend/replication/README                |   93 +
 src/backend/replication/TODO                  |  161 ++
 src/backend/replication/cset.c                |  280 +++
 src/backend/replication/gc_egcs.c             |  580 ++++++++
 src/backend/replication/gc_ensemble.c         |  823 +++++++++++
 src/backend/replication/gc_spread.c           |  432 ++++++
 src/backend/replication/gc_utils.c            |   93 +
 src/backend/replication/local.c               |  878 ++++++++++++
 src/backend/replication/manager.c             | 1859 ++++++++++++++++++++++++++
 src/backend/replication/recovery.c            | 1375 +++++++++++++++++++
 src/backend/replication/remote.c              |  463 ++++++
 src/backend/replication/utils.c               |  558 +++++++
 src/backend/storage/ipc/Makefile              |    4
 src/backend/storage/ipc/buffer.c              |  308 ++++
 src/backend/storage/ipc/imsg.c                |  369 +++++
 src/backend/storage/ipc/ipci.c                |   12
 src/backend/storage/ipc/procarray.c           |  106 +
 src/backend/storage/lmgr/lock.c               |    2
 src/backend/storage/lmgr/proc.c               |   70
 src/backend/tcop/postgres.c                   |  421 +++++
 src/backend/tcop/utility.c                    |    6
 src/backend/utils/cache/syscache.c            |    1
 src/backend/utils/init/globals.c              |   24
 src/backend/utils/init/miscinit.c             |    6
 src/backend/utils/init/postinit.c             |    4
 src/backend/utils/misc/check_guc              |    4
 src/backend/utils/misc/guc.c                  |   70
 src/backend/utils/misc/postgresql.conf.sample |   11
 src/include/Makefile                          |    4
 src/include/access/transam.h                  |    4
 src/include/access/xact.h                     |   10
 src/include/catalog/indexing.h                |    5
 src/include/catalog/pg_database.h             |   12
 src/include/catalog/pg_replication_gcs.h      |   60
 src/include/commands/dbcommands.h             |    2
 src/include/commands/replicacmds.h            |   22
 src/include/commands/sequence.h               |   16
 src/include/commands/variable.h               |    3
 src/include/executor/executor.h               |    7
 src/include/miscadmin.h                       |    4
 src/include/nodes/execnodes.h                 |    8
 src/include/nodes/nodes.h                     |    1
 src/include/nodes/parsenodes.h                |   12
 src/include/pg_config.h.in                    |    3
 src/include/replication/cset.h                |  121 +
 src/include/replication/gc.h                  |  215 +++
 src/include/replication/jobcache.h            |   30
 src/include/replication/manager.h             |  113 +
 src/include/replication/messages.h            |   51
 src/include/replication/recovery.h            |   67
 src/include/replication/replication.h         |  132 +
 src/include/replication/utils.h               |   71
 src/include/storage/buffer.h                  |   63
 src/include/storage/imsg.h                    |   83 +
 src/include/storage/proc.h                    |   20
 src/include/storage/procarray.h               |    8
 src/include/utils/guc_tables.h                |    4
 src/interfaces/ecpg/preproc/preproc.y         |   16
 src/test/regress/expected/sanity_check.out    |    3
 src/test/regress/expected/sequence.out        |   12
 98 files changed, 12777 insertions(+), 68 deletions(-)

Re: Postgres-R source code release

From
Devrim GÜNDÜZ
Date:
On Mon, 2008-07-14 at 17:42 +0200, Markus Wanner wrote:
> As much as I'd like to change that, I now think the best thing for the
> project itself is to open it up and release the source code. I'd very
> much like to get others aboard and turn Postgres-R into a real
> community  project again. Of course, the recent core statement about
> wanting an  integrated replication solution also influenced my
> decision, even though I realize that Postgres-R is way more ambitious
> path than a log shipping solution.
>
> As a first step towards a community project, I've cleaned up the code
> and tried to comment and document it as good as I can. You are welcome
> to download the latest patch from
> http://www.postgres-r.org/downloads/.

That's good news, really.

> In case you are wondering: I've got married and I'm glad to tell you
> that I changed my very German sounding name Schiltknecht to a more
> pronounceable one: my new surname is Wanner.

Congrats :)
--
Devrim GÜNDÜZ
devrim~gunduz.org, devrim~PostgreSQL.org, devrim.gunduz~linux.org.tr                  http://www.gunduz.org


Re: Postgres-R source code release

From
"Jonah H. Harris"
Date:
On Mon, Jul 14, 2008 at 11:42 AM, Markus Wanner
<markus.wanner@programmfabrik.de> wrote:
> As a first step towards a community project, I've cleaned up the code and
> tried to comment and document it as good as I can. You are welcome to
> download the latest patch from http://www.postgres-r.org/downloads/. It
> expects to be applied against today's CVS HEAD. You'll find a README and a
> pretty exhaustive TODO file under src/backend/replication/.

Awesome!

> In case you are wondering: I've got married

Congratulations!

-- 
Jonah H. Harris, Sr. Software Architect | phone: 732.331.1324
EnterpriseDB Corporation | fax: 732.331.1301
499 Thornall Street, 2nd Floor | jonah.harris@enterprisedb.com
Edison, NJ 08837 | http://www.enterprisedb.com/


Re: Postgres-R source code release

From
David Fetter
Date:
On Mon, Jul 14, 2008 at 05:42:21PM +0200, Markus Wanner wrote:
> Dear Hackers,
>
> it has been two years, since I've presented my work on Postgres-R for  
> the first time to a broader audience in Toronto. I continued maintaining  
> that code in my spare time, but to be honest, I didn't have much time  
> for it.
>
> As much as I'd like to change that, I now think the best thing for the  
> project itself is to open it up and release the source code. I'd very  
> much like to get others aboard and turn Postgres-R into a real community  
> project again. Of course, the recent core statement about wanting an  
> integrated replication solution also influenced my decision, even though  
> I realize that Postgres-R is way more ambitious path than a log shipping  
> solution.
>
> As a first step towards a community project, I've cleaned up the code  
> and tried to comment and document it as good as I can. You are welcome  
> to download the latest patch from http://www.postgres-r.org/downloads/.  
> It expects to be applied against today's CVS HEAD. You'll find a README  
> and a pretty exhaustive TODO file under src/backend/replication/.

Would you mind if I were to make a git branch for it on
<http://git.postgresql.org/> ?

> In case you are wondering: I've got married

Congratulations!

Cheers,
David.
-- 
David Fetter <david@fetter.org> http://fetter.org/
Phone: +1 415 235 3778  AIM: dfetter666  Yahoo!: dfetter
Skype: davidfetter      XMPP: david.fetter@gmail.com

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate


Re: Postgres-R source code release

From
Alvaro Herrera
Date:
David Fetter wrote:
> On Mon, Jul 14, 2008 at 05:42:21PM +0200, Markus Wanner wrote:

> Would you mind if I were to make a git branch for it on
> <http://git.postgresql.org/> ?

That's very likely wasted effort, since obviously Markus has got a
Monotone tree somewhere ...

-- 
Alvaro Herrera                                http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support


Re: Postgres-R source code release

From
David Fetter
Date:
On Mon, Jul 14, 2008 at 05:35:28PM -0400, Alvaro Herrera wrote:
> David Fetter wrote:
> > On Mon, Jul 14, 2008 at 05:42:21PM +0200, Markus Wanner wrote:
> 
> > Would you mind if I were to make a git branch for it on
> > <http://git.postgresql.org/> ?
> 
> That's very likely wasted effort, since obviously Markus has got a
> Monotone tree somewhere ...

I'm curious as to your motive behind continual deriding any effort to
get git going on branches.  Do you have some other alternative in mind?

Cheers,
David.
-- 
David Fetter <david@fetter.org> http://fetter.org/
Phone: +1 415 235 3778  AIM: dfetter666  Yahoo!: dfetter
Skype: davidfetter      XMPP: david.fetter@gmail.com

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate


Re: Postgres-R source code release

From
Alvaro Herrera
Date:
David Fetter wrote:
> On Mon, Jul 14, 2008 at 05:35:28PM -0400, Alvaro Herrera wrote:
> > David Fetter wrote:
> > > On Mon, Jul 14, 2008 at 05:42:21PM +0200, Markus Wanner wrote:
> > 
> > > Would you mind if I were to make a git branch for it on
> > > <http://git.postgresql.org/> ?
> > 
> > That's very likely wasted effort, since obviously Markus has got a
> > Monotone tree somewhere ...
> 
> I'm curious as to your motive behind continual deriding any effort to
> get git going on branches.  Do you have some other alternative in mind?

That's not my intention.  It's just that the two cases that have popped
up are not useful.  The with-recursive one because upstream does not use
it.  If that can be solved, then I have no objection and I think it
would be a good thing.  This one because it would be messy.  I think the
way to go here is to have Markus open up his Monotone repo, or convince
him to migrate it to Git, but I really doubt that's ever going to
happen.

-- 
Alvaro Herrera                                http://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.


Re: Postgres-R source code release

From
"Joshua D. Drake"
Date:
On Mon, 2008-07-14 at 14:41 -0700, David Fetter wrote:
> On Mon, Jul 14, 2008 at 05:35:28PM -0400, Alvaro Herrera wrote:
> > David Fetter wrote:
> > > On Mon, Jul 14, 2008 at 05:42:21PM +0200, Markus Wanner wrote:
> > 
> > > Would you mind if I were to make a git branch for it on
> > > <http://git.postgresql.org/> ?
> > 
> > That's very likely wasted effort, since obviously Markus has got a
> > Monotone tree somewhere ...
> 
> I'm curious as to your motive behind continual deriding any effort to
> get git going on branches.  Do you have some other alternative in mind?

Obviously monotone, but I think the point is, Git gives them nothing
Monotone doesn't. Of course, I am personally happy with SVN but hey :P

Joshua D. Drake

-- 
The PostgreSQL Company since 1997: http://www.commandprompt.com/ 
PostgreSQL Community Conference: http://www.postgresqlconference.org/
United States PostgreSQL Association: http://www.postgresql.us/
Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate





Re: Postgres-R source code release

From
David Fetter
Date:
On Mon, Jul 14, 2008 at 04:22:47PM -0700, Joshua D. Drake wrote:
> On Mon, 2008-07-14 at 14:41 -0700, David Fetter wrote:
> > On Mon, Jul 14, 2008 at 05:35:28PM -0400, Alvaro Herrera wrote:
> > > David Fetter wrote:
> > > > On Mon, Jul 14, 2008 at 05:42:21PM +0200, Markus Wanner wrote:
> > > 
> > > > Would you mind if I were to make a git branch for it on
> > > > <http://git.postgresql.org/> ?
> > > 
> > > That's very likely wasted effort, since obviously Markus has got
> > > a Monotone tree somewhere ...
> > 
> > I'm curious as to your motive behind continual deriding any effort
> > to get git going on branches.  Do you have some other alternative
> > in mind?
> 
> Obviously monotone, but I think the point is, Git gives them nothing
> Monotone doesn't.

Apart from being able to interoperate with CVS, you mean? ;)

> Of course, I am personally happy with SVN but hey :P

You can't have tried a merge in SVN if that's so :P :P

Cheers,
David.
-- 
David Fetter <david@fetter.org> http://fetter.org/
Phone: +1 415 235 3778  AIM: dfetter666  Yahoo!: dfetter
Skype: davidfetter      XMPP: david.fetter@gmail.com

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate


Re: Postgres-R source code release

From
"David E. Wheeler"
Date:
On Jul 14, 2008, at 16:54, David Fetter wrote:

>> Of course, I am personally happy with SVN but hey :P
>
> You can't have tried a merge in SVN if that's so :P :P

Those of us who have been doing it for years, in CVS and in SVN,  
aren't too worried about it.

Best,

David


Re: Postgres-R source code release

From
"Andrej Ricnik-Bay"
Date:
On 15/07/2008, David E. Wheeler <david@kineticode.com> wrote:
> > > Of course, I am personally happy with SVN but hey :P
> >
> > You can't have tried a merge in SVN if that's so :P :P
>
>  Those of us who have been doing it for years, in CVS and in SVN, aren't too
> worried about it.
>

Follow the sandal! :D



-- 
Please don't top post, and don't use HTML e-Mail :}  Make your quotes concise.

http://www.american.edu/econ/notes/htmlmail.htm


Re: Postgres-R source code release

From
Markus Wanner
Date:
Hi,

Alvaro Herrera wrote:
> I think the
> way to go here is to have Markus open up his Monotone repo, or convince
> him to migrate it to Git, but I really doubt that's ever going to
> happen.

He he... good guess ;-)

However, as much as I personally like monotone and as much as I dislike
git for being a bad rip off of monotone, git certainly has its merits.
It has a much larger users base and is faster than monotone for some 
operations.

Having used subversion, CVS and monotone for Postgres-R, I think I'm now
giving git a try. I'm about to setup the necessary repositories, but I'm 
still having a hard time grokking this tool set. (And yes, I'm missing 
some features compared to monotone. In our case, the most obvious one is 
'mtn diff --context' ;-) )

Regards

Markus


Re: Postgres-R source code release

From
Markus Wanner
Date:
Hi,

David Fetter wrote:
> Would you mind if I were to make a git branch for it on
> <http://git.postgresql.org/> ?

I've set up a git-daemon with the Postgres-R patch here:

git://postgres-r.org/repo

Since it's a distributed VCS, you should be able to mirror that to 
git.postgtresql.org somehow (if you figure out how, please tell me!).

Please note that I'm still struggling with git and I cannot promise to 
keep using it.

Regards

Markus



Re: Postgres-R source code release

From
David Fetter
Date:
On Wed, Jul 16, 2008 at 09:35:28PM +0200, Markus Schiltknecht wrote:
> Hi,
>
> David Fetter wrote:
>> Would you mind if I were to make a git branch for it on
>> <http://git.postgresql.org/> ?
>
> I've set up a git-daemon with the Postgres-R patch here:
>
> git://postgres-r.org/repo
>
> Since it's a distributed VCS, you should be able to mirror that to  
> git.postgtresql.org somehow (if you figure out how, please tell me!).

I've merged the latest Postgres in.  Care to see whether it runs?
http://git.postgresql.org/?p=~davidfetter/pgr/.git;a=summary

> Please note that I'm still struggling with git and I cannot promise
> to  keep using it.

I'm struggling, too, but the cheapness of experimenting is making it
easier and easier :)

Cheers,
David.
-- 
David Fetter <david@fetter.org> http://fetter.org/
Phone: +1 415 235 3778  AIM: dfetter666  Yahoo!: dfetter
Skype: davidfetter      XMPP: david.fetter@gmail.com

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate