Thread: Two-phase commit for 8.1

Two-phase commit for 8.1

From
Heikki Linnakangas
Date:
Hi,

Now that we got 8.0 out of the door, I'm submitting my two-phase commit patch 
again for discussion.

http://www.hut.fi/~hlinnaka/pgsql/

Do we want it in 8.1, if we want a short development cycle? It needs a new 
pg_twophase subdirectory, and it introduces a new system view, so I guess it 
requires an initdb (or pg_upgrade).

Any comments on the implementation or the new commands?

I would appreciate help testing the JDBC driver with an application server that 
does XA recovery properly.

- Heikki


Re: Two-phase commit for 8.1

From
"Marc G. Fournier"
Date:
If the patch is ready to be committed early in the cycle, I'd say most 
definitely ... just depends on how late in the cycle its ready ...

I *believe* that 8.1, we're looking at a 2mo cycle before beta, so figure 
beta for ~April 1st (no april fools jokes, eh?) ...

On Wed, 19 Jan 2005, Heikki Linnakangas wrote:

> Hi,
>
> Now that we got 8.0 out of the door, I'm submitting my two-phase commit patch 
> again for discussion.
>
> http://www.hut.fi/~hlinnaka/pgsql/
>
> Do we want it in 8.1, if we want a short development cycle? It needs a new 
> pg_twophase subdirectory, and it introduces a new system view, so I guess it 
> requires an initdb (or pg_upgrade).
>
> Any comments on the implementation or the new commands?
>
> I would appreciate help testing the JDBC driver with an application server 
> that does XA recovery properly.
>
> - Heikki
>
> ---------------------------(end of broadcast)---------------------------
> TIP 2: you can get off all lists at once with the unregister command
>   (send "unregister YourEmailAddressHere" to majordomo@postgresql.org)
>

----
Marc G. Fournier           Hub.Org Networking Services (http://www.hub.org)
Email: scrappy@hub.org           Yahoo!: yscrappy              ICQ: 7615664


Re: Two-phase commit for 8.1

From
Tom Lane
Date:
"Marc G. Fournier" <scrappy@postgresql.org> writes:
> If the patch is ready to be committed early in the cycle, I'd say most 
> definitely ... just depends on how late in the cycle its ready ...

My recollection is that it's quite far from being complete.  I had hoped
to spend some time during the 8.1 cycle helping Heikki finish it up,
but if we stick to the 2-month-dev-cycle idea I'm afraid there's no way
it'll be done in time.  I thought that "some time" would probably amount
to a solid man-month or so, and there's no way I can spend half my time
on just one feature for this cycle.

If Heikki wants this in for 8.1, the right thing to do is vote against
the short-dev-cycle idea.  But we need a plausible answer about what to
do about ARC to make that credible...
        regards, tom lane


Re: Two-phase commit for 8.1

From
Christopher Kings-Lynne
Date:
> If the patch is ready to be committed early in the cycle, I'd say most 
> definitely ... just depends on how late in the cycle its ready ...
> 
> I *believe* that 8.1, we're looking at a 2mo cycle before beta, so 
> figure beta for ~April 1st (no april fools jokes, eh?) ...

You guys are crazy :)  We haven't had a release cycle less than a year 
in many, many releases :)

Chris


Re: Two-phase commit for 8.1

From
Robert Treat
Date:
On Thursday 20 January 2005 04:16, Christopher Kings-Lynne wrote:
> > If the patch is ready to be committed early in the cycle, I'd say most
> > definitely ... just depends on how late in the cycle its ready ...
> >
> > I *believe* that 8.1, we're looking at a 2mo cycle before beta, so
> > figure beta for ~April 1st (no april fools jokes, eh?) ...
>
> You guys are crazy :)  We haven't had a release cycle less than a year
> in many, many releases :)
>

If ARC is deemed a serious enough problem that we need to address it *now*, 
then I think we should do a 2 month cycle where core focus will be on putting 
in an ARC replacement and allowing only changes that do not require an 
initdb.  If we stick to that we can do a new release in probably 3-4 months 
and I think that will be acceptable as long as dump/reload is not required 
for upgrade.  I still think it might be worth contacting IBM and asking them 
if thier intention is to enforce the ARC patent if it is granted before 
pushing forward with that plan, but it seems like a good course of action to 
follow just in case.  If others felt strongly, we could probably start an 8.2 
branch right now as well and put someone like Neil in charge of keeping 8.2 
up to date with 8.1 changes while proceeding forward with other new whizbang 
functionality that requires initdb (I pick Neil cause iirc he has some initdb 
requiring changes planned for development, but it could be someone else).

-- 
Robert Treat
Build A Brighter Lamp :: Linux Apache {middleware} PostgreSQL


Re: Two-phase commit for 8.1

From
Kenneth Marshall
Date:
On Wed, Jan 19, 2005 at 07:42:03PM -0500, Tom Lane wrote:
> "Marc G. Fournier" <scrappy@postgresql.org> writes:
> > If the patch is ready to be committed early in the cycle, I'd say most 
> > definitely ... just depends on how late in the cycle its ready ...
> 
> My recollection is that it's quite far from being complete.  I had hoped
> to spend some time during the 8.1 cycle helping Heikki finish it up,
> but if we stick to the 2-month-dev-cycle idea I'm afraid there's no way
> it'll be done in time.  I thought that "some time" would probably amount
> to a solid man-month or so, and there's no way I can spend half my time
> on just one feature for this cycle.
> 
> If Heikki wants this in for 8.1, the right thing to do is vote against
> the short-dev-cycle idea.  But we need a plausible answer about what to
> do about ARC to make that credible...
> 

I think the idea of making a buffer management algorithm API and then
preparing a simple LRU algorithm to have ready to plug in if the ARC
patent is granted would be doable in a short development cycle. Then
we can take advantage of as well as test other algorithms more easily.

Ken


Re: Two-phase commit for 8.1

From
Heikki Linnakangas
Date:
On Wed, 19 Jan 2005, Tom Lane wrote:

> "Marc G. Fournier" <scrappy@postgresql.org> writes:
>> If the patch is ready to be committed early in the cycle, I'd say most
>> definitely ... just depends on how late in the cycle its ready ...
>
> My recollection is that it's quite far from being complete.  I had hoped
> to spend some time during the 8.1 cycle helping Heikki finish it up,
> but if we stick to the 2-month-dev-cycle idea I'm afraid there's no way
> it'll be done in time.  I thought that "some time" would probably amount
> to a solid man-month or so, and there's no way I can spend half my time
> on just one feature for this cycle.
>
> If Heikki wants this in for 8.1, the right thing to do is vote against
> the short-dev-cycle idea.  But we need a plausible answer about what to
> do about ARC to make that credible...

I'm not sure what I want.

If the 8.1 cycle really is a short one, say 3 months, then I have no 
problem waiting for 8.2. But we have a very bad track record regarding 
short-dev-cycles. I honestly don't believe we can get 8.1 released before 
July.

- Heikki


Re: Two-phase commit for 8.1

From
Hans-Jürgen Schönig
Date:
Heikki,

What is still missing to complete the 2PC patch?.
Regards,
    Hans


Heikki Linnakangas wrote:
> On Wed, 19 Jan 2005, Tom Lane wrote:
> 
>> "Marc G. Fournier" <scrappy@postgresql.org> writes:
>>
>>> If the patch is ready to be committed early in the cycle, I'd say most
>>> definitely ... just depends on how late in the cycle its ready ...
>>
>>
>> My recollection is that it's quite far from being complete.  I had hoped
>> to spend some time during the 8.1 cycle helping Heikki finish it up,
>> but if we stick to the 2-month-dev-cycle idea I'm afraid there's no way
>> it'll be done in time.  I thought that "some time" would probably amount
>> to a solid man-month or so, and there's no way I can spend half my time
>> on just one feature for this cycle.
>>
>> If Heikki wants this in for 8.1, the right thing to do is vote against
>> the short-dev-cycle idea.  But we need a plausible answer about what to
>> do about ARC to make that credible...
> 
> 
> I'm not sure what I want.
> 
> If the 8.1 cycle really is a short one, say 3 months, then I have no 
> problem waiting for 8.2. But we have a very bad track record regarding 
> short-dev-cycles. I honestly don't believe we can get 8.1 released 
> before July.
> 
> - Heikki
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 8: explain analyze is your friend


-- 
Cybertec Geschwinde u Schoenig
Schoengrabern 134, A-2020 Hollabrunn, Austria
Tel: +43/660/816 40 77
www.cybertec.at, www.postgresql.at



Re: Two-phase commit for 8.1

From
Heikki Linnakangas
Date:
On Sun, 23 Jan 2005, Hans-Jürgen Schönig wrote:

> Heikki,
>
> What is still missing to complete the 2PC patch?.

Here's my TODO on things that need to be done:
    * large objects    * guc variables    * notify/listen

Large objects and notify/listen should be quite straightforward. GUC 
variables need some thinking, but shouldn't be much work.

As the patch gets more attention, I'm sure more issues will come up.

- Heikki


Re: Two-phase commit for 8.1

From
Alvaro Herrera
Date:
On Sun, Jan 23, 2005 at 01:37:30PM +0200, Heikki Linnakangas wrote:

> As the patch gets more attention, I'm sure more issues will come up.

I see the changes to the lock manager are huge.  Can you explain what's
the idea behind those?  Do you release the locks and then reacquire
them, or do you reassign them to a pseudo process?  Are there
possibilities of deadlock somewhere?

-- 
Alvaro Herrera (<alvherre[@]dcc.uchile.cl>)
Thou shalt study thy libraries and strive not to reinvent them without
cause, that thy code may be short and readable and thy days pleasant
and productive. (7th Commandment for C Programmers)


Re: Two-phase commit for 8.1

From
Heikki Linnakangas
Date:
On Sun, 23 Jan 2005, Alvaro Herrera wrote:

> On Sun, Jan 23, 2005 at 01:37:30PM +0200, Heikki Linnakangas wrote:
>
>> As the patch gets more attention, I'm sure more issues will come up.

> I see the changes to the lock manager are huge.  Can you explain what's
> the idea behind those?  Do you release the locks and then reacquire
> them, or do you reassign them to a pseudo process?

I reassign them to a pseudo process (persistedLocksProc).

Much of the changes in lock.c are just about moving code around. 
Some copy-paste code has been put in functions. LockAcquire has been
changed to take PGPROC as an argument, so that locks can be acquired on 
behalf of the pseudo process.

Then there's completely new code for persisting locks on PREPARE 
TRANSACTION and reacquiring them on recovery.

If it helps, I could try to split it into two patches, one with code 
rearrangements that don't change current behaviour, and then the actual 
2PC stuff on top of that.

> Are there possibilities of deadlock somewhere?

Not that I know of.

- Heikki


Re: Two-phase commit for 8.1

From
Alvaro Herrera
Date:
On Sun, Jan 23, 2005 at 07:32:55PM +0200, Heikki Linnakangas wrote:

> If it helps, I could try to split it into two patches, one with code 
> rearrangements that don't change current behaviour, and then the actual 
> 2PC stuff on top of that.

I think that'd be a good idea, because such a patch could be merged
right now, and the actual 2PC stuff would be smaller and easier to
review.

You'd only need a committer to actually commit the initial patch ...

-- 
Alvaro Herrera (<alvherre[@]dcc.uchile.cl>)
"No hay cielo posible sin hundir nuestras raícesen la profundidad de la tierra"                        (Malucha Pinto)