Thread: damage control mode

On Thu, Jan 7, 2010 at 4:23 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Robert Haas <robertmhaas@gmail.com> writes:
>> On Thu, Jan 7, 2010 at 3:36 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>>> Robert Haas <robertmhaas@gmail.com> writes:
>>>> I am tempted to say we should clamp down and go into damage control
>>>> mode sooner rather than later.
>>>
>>> The more I see of the HS patch, the more I think the same.  But my
>>> proposal for "damage control mode" would be to immediately punt
>>> everything else to the next release and focus our energies exclusively
>>> on HS *and* SR.
>
>> Hmm.  There's something to what you say, but what about the people who
>> were expecting their patches to be reviewed and perhaps committed in
>> the forthcoming CommitFest.  I proposed a schedule for this release
>> that involved only three CommitFests and it was rejected, so it seems
>> a bit unfair to pull the rug out from under people at the eleventh
>> hour.  Will we lose developers if we do this?
>
> Well, I think we should put at least some effort into clearing out
> the underbrush.  There's a lot of pretty small stuff in the January CF
> list that I think we could go through in a short time.  The biggies,
> IMO, are the ones you noted:
>
>> Unfortunately, there are some patches that I probably will not feel
>> confident to commit without your input - in particular, writeable
>> CTEs, listen/notify, more frame options in window functions -
>
> and Teodor's GIN/GIST stuff.  If we feel that we are in schedule
> trouble I think that bumping these to the next release is by far
> the sanest response.  Bumping SR so we can get these in would be
> completely misguided.

OK, we have a proposal on the table to bump some patches from this
CommitFest to free up more committer resources, particularly Tom, to
work on Hot Standby and Streaming Replication and attempt to
accelerate the process of getting 8.5 out the door.  This proposal
needs input from the community.  The affected patches are:

- Listen/Notify Rewrite.
- Writeable CTEs.
- more frame options for window functions
- knngist
- rbtree

Tom wasn't specific about which of the GIN/GIST patches he was
discussing, but I'm excluding point_ops from this list because I have
already reviewed it and believe that it requires only trivial changes
prior to commit.

I believe that it is certainly fair to bump the last three of these
patches, because they are all large patches which have been submitted
for the first time at the end of the development cycle.  I previously
suggested a rule of thumb that any patch which, when considered as a
unified diff, had a total number of lines added and lines removed >
1000, would not be considered for the last CommitFest unless it had
been submitted to the second-to-last CommitFest.  That rule would
apply to all three of these patches (though rbtree only just barely)
and I think it makes sense to go ahead and apply it, postponing all of
these to 8.6.

The decision to preemptively bump listen/notify rewrite or writeable
CTEs would be somewhat more painful to me because I do feel that the
patch authors have basically followed the rules - both have been
submitted and reviewed previously and are resubmitted here with
improvements coming out of those prior reviews.  On the other hand, we
don't have infinite resources, and Streaming Replication is a
big-ticket feature whose patch author has ALSO followed the rules - in
fact, even moreso - I don't believe it's received a full review since
being submitted, after a substantial rewrite, to the SEPTEMBER
CommitFest.  So I could go either way on this one.

It should be noted that the decision to bump these patches does not
guarantee that SR will be part of 8.5, though it should improve our
chances.  It also does not guarantee that the release will happen "on
time", though it will presumably help with that, too.

Votes?

...Robert

On Thu, Jan 07, 2010 at 08:57:15PM -0500, Robert Haas wrote:
> On Thu, Jan 7, 2010 at 4:23 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> > Robert Haas <robertmhaas@gmail.com> writes:
> >> On Thu, Jan 7, 2010 at 3:36 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> >>> Robert Haas <robertmhaas@gmail.com> writes:
> >>>> I am tempted to say we should clamp down and go into damage control
> >>>> mode sooner rather than later.
> >>>
> >>> The more I see of the HS patch, the more I think the same.  But my
> >>> proposal for "damage control mode" would be to immediately punt
> >>> everything else to the next release and focus our energies exclusively
> >>> on HS *and* SR.
> >
> >> Hmm.  There's something to what you say, but what about the people who
> >> were expecting their patches to be reviewed and perhaps committed in
> >> the forthcoming CommitFest.  I proposed a schedule for this release
> >> that involved only three CommitFests and it was rejected, so it seems
> >> a bit unfair to pull the rug out from under people at the eleventh
> >> hour.  Will we lose developers if we do this?
> >
> > Well, I think we should put at least some effort into clearing out
> > the underbrush.  There's a lot of pretty small stuff in the January CF
> > list that I think we could go through in a short time.  The biggies,
> > IMO, are the ones you noted:
> >
> >> Unfortunately, there are some patches that I probably will not feel
> >> confident to commit without your input - in particular, writeable
> >> CTEs, listen/notify, more frame options in window functions -
> >
> > and Teodor's GIN/GIST stuff.  If we feel that we are in schedule
> > trouble I think that bumping these to the next release is by far
> > the sanest response.  Bumping SR so we can get these in would be
> > completely misguided.
> 
> OK, we have a proposal on the table to bump some patches from this
> CommitFest to free up more committer resources, particularly Tom, to
> work on Hot Standby and Streaming Replication and attempt to
> accelerate the process of getting 8.5 out the door.  This proposal
> needs input from the community.  The affected patches are:

I don't see any need to thump the panic button today.

If we *must* have SR and it's not in by the 15th, let's do another
Commitfest rather than jack the people who played by the rules.

Cheers,
David.
-- 
David Fetter <david@fetter.org> http://fetter.org/
Phone: +1 415 235 3778  AIM: dfetter666  Yahoo!: dfetter
Skype: davidfetter      XMPP: david.fetter@gmail.com
iCal: webcal://www.tripit.com/feed/ical/people/david74/tripit.ics

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

Robert Haas wrote:
>
> OK, we have a proposal on the table to bump some patches from this
> CommitFest to free up more committer resources, particularly Tom, to
> work on Hot Standby and Streaming Replication and attempt to
> accelerate the process of getting 8.5 out the door.  This proposal
> needs input from the community.  The affected patches are:
>
> - Listen/Notify Rewrite.
> - Writeable CTEs.
> - more frame options for window functions
> - knngist
> - rbtree
>   

This strikes me as quite premature. Heiki just said he expects to have 
SR committed next week.

Maybe we need a copy of the Hitchhiker's Guide to the Galaxy, or at 
least its front cover 
(<http://myhodgepodge.files.wordpress.com/2008/07/hitchhikers_guide_to_galaxy.jpg>)

cheers

andrew

* David Fetter (david@fetter.org) wrote:
> > OK, we have a proposal on the table to bump some patches from this
> > CommitFest to free up more committer resources, particularly Tom, to
> > work on Hot Standby and Streaming Replication and attempt to
> > accelerate the process of getting 8.5 out the door.  This proposal
> > needs input from the community.  The affected patches are:
>
> I don't see any need to thump the panic button today.
>
> If we *must* have SR and it's not in by the 15th, let's do another
> Commitfest rather than jack the people who played by the rules.

Playing by the rules isn't a guarantee, sorry.  We have to prioritize
at some point or we'll never get it done.  It seems like typically we do
that when we're past the point where we had originally planned to
release and only do it because of that pressure.  I would have flipped
the priority of the two top items (I'd rather have writeable CTE than a
new listen/notify), but that doesn't change that I think they're under
SR.

I'm gonna have to +1 bumping the lower priority items in favor of having
an on-time release.  To be honest, and I don't intend this as a slight
to anyone, but I'm already worried that just getting HS into a state
where everyone is comfortable releasing it to the wild and having users
trust their data to 8.5 is going to be a serious effort between now and
release.

Perhaps if we discover HS and SR go in alot more easily than expected we
could revisit the decision to bump things, but I don't like encouraging
false hope.  Would things really happen that way?  I tend to doubt it,
since I think after we get HS and SR done there's going to be an
appropriate push for beta.

Just my 2c from reading the list and history.  I'm happy to be wrong.
Thanks,
    Stephen

On Thu, 2010-01-07 at 20:57 -0500, Robert Haas wrote:
> - Listen/Notify Rewrite.
> - Writeable CTEs.
...
> Votes?

I'm not qualified to vote on how other people spend their time, but here
are my thoughts:

SR was submitted quite some time ago, so I don't see it as breaking the
rules to put it first in line. If we never give the big features the
serious attention required, then they will never get committed. However,
that's easy for me to say, because my feature made it; and we shouldn't
dismiss the frustration of following the rules and still missing the
boat.

Aside: I'll take this alarm as a very strong hint that I shouldn't push
the "range types" any more until the next development cycle.
Particularly because Tom is one of the people with opinions about it, so
I don't want to distract him from features submitted several commitfests
ago.

Regards,Jeff Davis

On Thu, Jan 7, 2010 at 9:11 PM, Andrew Dunstan <andrew@dunslane.net> wrote:
> This strikes me as quite premature. Heiki just said he expects to have SR
> committed next week.

Getting it committed is not what I'm worried about.  What I'm
concerned about is Tom's statement that he believes that HS is not
close to being in a state where it is stable enough to go to beta, and
the possibility that other large patches committed during this
CommitFest might have similar issues.

> Maybe we need a copy of the Hitchhiker's Guide to the Galaxy, or at least
> its front cover
> (<http://myhodgepodge.files.wordpress.com/2008/07/hitchhikers_guide_to_galaxy.jpg>)

I like to save my panicking for things that are worth it, which the
PostgreSQL release schedule is not.  But I do think that is worth an
honest discussion of what is possible.  Do you feel we can commit six
large patches in the next month, stabilize all of them plus Hot
Standby, and put out a release in June?  If not, would you prefer to
slip some patches or slip the schedule?

...Robert

Robert Haas <robertmhaas@gmail.com> writes:
> OK, we have a proposal on the table to bump some patches from this
> CommitFest to free up more committer resources, particularly Tom, to
> work on Hot Standby and Streaming Replication and attempt to
> accelerate the process of getting 8.5 out the door.  This proposal
> needs input from the community.  The affected patches are:

> - Listen/Notify Rewrite.
> - Writeable CTEs.
> - more frame options for window functions
> - knngist
> - rbtree

Looking at this list again, it strikes me that the listen/notify rewrite
might need to go in so that we have a sane framework for listen/notify
with HS.  Treating pg_listener as a replicatable table isn't sane at
all, whereas with a message-driven notify mechanism we have at least got
the possibility of shipping the messages to the standby (via WAL) and
having listeners there.  I don't want to say we'd actually implement
that in 8.5, but shipping pg_listener tuple updates is just completely
nuts.

The other four things have no apparent connection to HS/SR so I think
they could be punted without creating technical issues.  Whether this
is really necessary from a schedule viewpoint is not clear yet.

My thought about it would be to put these four on the back burner;
not necessarily bounce them right away, but not put any effort into
them until we have dealt with the other stuff in the January CF.
At that point we should have a feel for where we are schedule-wise,
and in particular we'll know whether SR is in or not ;-)
        regards, tom lane

On Thu, Jan 7, 2010 at 10:20 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Robert Haas <robertmhaas@gmail.com> writes:
>> OK, we have a proposal on the table to bump some patches from this
>> CommitFest to free up more committer resources, particularly Tom, to
>> work on Hot Standby and Streaming Replication and attempt to
>> accelerate the process of getting 8.5 out the door.  This proposal
>> needs input from the community.  The affected patches are:
>
>> - Listen/Notify Rewrite.
>> - Writeable CTEs.
>> - more frame options for window functions
>> - knngist
>> - rbtree
>
> Looking at this list again, it strikes me that the listen/notify rewrite
> might need to go in so that we have a sane framework for listen/notify
> with HS.  Treating pg_listener as a replicatable table isn't sane at
> all, whereas with a message-driven notify mechanism we have at least got
> the possibility of shipping the messages to the standby (via WAL) and
> having listeners there.  I don't want to say we'd actually implement
> that in 8.5, but shipping pg_listener tuple updates is just completely
> nuts.

It's also related to this open issue, so possibly we could kill two
birds with one stone (although I guess that would still leave the
problem of what to do in the back branches).

http://archives.postgresql.org/pgsql-bugs/2009-12/msg00274.php

> The other four things have no apparent connection to HS/SR so I think
> they could be punted without creating technical issues.  Whether this
> is really necessary from a schedule viewpoint is not clear yet.
>
> My thought about it would be to put these four on the back burner;
> not necessarily bounce them right away, but not put any effort into
> them until we have dealt with the other stuff in the January CF.
> At that point we should have a feel for where we are schedule-wise,
> and in particular we'll know whether SR is in or not ;-)

That seems wishy-washy.  If we're going to have any chance of getting
these patches in, we have to give the patch authors good feedback
early in the CommitFest so that they have time to make the necessary
revisions before the end of the CommitFest.  If we think we can swing
it, I'm happy to handle these patches in the normal way; I'm also
happy to say we'll review them all but not commit them for fear of
destabilizing the tree; or we can punt them altogether.  We can also
pick and choose.  But saying we're going to postpone dealing with them
doesn't seem to me to solve anything.

I also think that postponement is not going to buy us very much time
anyway.   Most of them are pretty small.

If SR is in, does that militate toward bumping the patches (because
now we have more potential bugs to fix) or against it (because now you
don't have to help make it committable)?

...Robert

Robert Haas <robertmhaas@gmail.com> writes:
> On Thu, Jan 7, 2010 at 10:20 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> Looking at this list again, it strikes me that the listen/notify rewrite
>> might need to go in so that we have a sane framework for listen/notify
>> with HS.

> It's also related to this open issue, so possibly we could kill two
> birds with one stone (although I guess that would still leave the
> problem of what to do in the back branches).
> http://archives.postgresql.org/pgsql-bugs/2009-12/msg00274.php

Uh, no, AFAICS that is an independent problem.  The actual bug is that
our Windows signal implementation can lose signals.  That spells trouble
in all kinds of contexts, not only NOTIFY.

>> My thought about it would be to put these four on the back burner;

> That seems wishy-washy.

Fair enough ;-).  But I don't feel a need to make a decision now,
either.  We can at least wait a week and see if Heikki gets SR
committed.
        regards, tom lane

On Thu, Jan 7, 2010 at 10:51 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Fair enough ;-).  But I don't feel a need to make a decision now,
> either.  We can at least wait a week and see if Heikki gets SR
> committed.

OK.

...Robert

Robert Haas wrote:
> If we're going to have any chance of getting
> these patches in, we have to give the patch authors good feedback
> early in the CommitFest so that they have time to make the necessary
> revisions before the end of the CommitFest.  If we think we can swing
> it, I'm happy to handle these patches in the normal way; I'm also
> happy to say we'll review them all but not commit them for fear of
> destabilizing the tree; or we can punt them altogether.

Presuming enough reviewers (which should be the case this time given the 
expectation that submitters also review), the suggested pacing here now 
has every patch passing through a round of review and potentially one 
update within ten days.  Given that, I'm not sure why you're looking to 
special case anything here.  Give everybody a fair initial run, just 
with the reminder that the bar for "ready for committer" is higher than 
normal on this last CF, which people have certainly gotten some warning 
of.  If your patch smells funny at all or seems like it will take a lot 
of work from a committer to apply, it's out.  Giving someone a review 
but then telling them "it looks good, but we just don't want to commit 
it right now" is more fair than not getting that author's patch a review 
at all, right?  So why bounce them prematurely?

-- 
Greg Smith    2ndQuadrant   Baltimore, MD
PostgreSQL Training, Services and Support
greg@2ndQuadrant.com  www.2ndQuadrant.com

David Fetter <david@fetter.org> writes:
> If we *must* have SR and it's not in by the 15th, let's do another
> Commitfest rather than jack the people who played by the rules.

If we do add another Commitfest what we do is exactly jacking people who
played by the rules. Because all those patches that are already part of
alpha3 have been worked on by people expecting a 4 CF development cycle,
and adjusted their agenda, and want a mid-year release.

Now, I'll second Greg Smith and Tom here, in that I think we need to run
the last commitfest as usual, knowing that the outcome of the commitfest
for any given patch is not "it made it" but "we reviewed it". It's still
right for the project to bump a patch on resources ground rather than on
technical merit, at the end of the commitfest.

Why we can do it this way is because we're not starving on
reviewers. We're starving on commiters time. And seeing this:
 https://commitfest.postgresql.org/action/commitfest_view?id=5
 Status Summary. Needs Review: 19, Waiting on Author: 5, Ready for Committer: 2, Committed: 9, Returned with Feedback:
4.Total: 39.
 

I don't see any reason not to consider all the 24 patches requiring our
attention.

Regards,
-- 
dim

On Fri, Jan 8, 2010 at 10:02, Dimitri Fontaine <dfontaine@hi-media.com> wrote:
> David Fetter <david@fetter.org> writes:
>> If we *must* have SR and it's not in by the 15th, let's do another
>> Commitfest rather than jack the people who played by the rules.
>
> If we do add another Commitfest what we do is exactly jacking people who
> played by the rules. Because all those patches that are already part of
> alpha3 have been worked on by people expecting a 4 CF development cycle,
> and adjusted their agenda, and want a mid-year release.
>
> Now, I'll second Greg Smith and Tom here, in that I think we need to run
> the last commitfest as usual, knowing that the outcome of the commitfest
> for any given patch is not "it made it" but "we reviewed it". It's still
> right for the project to bump a patch on resources ground rather than on
> technical merit, at the end of the commitfest.

+1.

> Why we can do it this way is because we're not starving on
> reviewers. We're starving on commiters time. And seeing this:

Well, we're actually somewhat starving on senior reviewers as well.
That can take on things like the index patches, Writable CTE or SR.
We're not starving on reviewers for small-to-medium patches.


-- Magnus HaganderMe: http://www.hagander.net/Work: http://www.redpill-linpro.com/

Magnus Hagander <magnus@hagander.net> writes:
>> Why we can do it this way is because we're not starving on
>> reviewers. We're starving on commiters time. And seeing this:
>
> Well, we're actually somewhat starving on senior reviewers as well.
> That can take on things like the index patches, Writable CTE or SR.
> We're not starving on reviewers for small-to-medium patches.

We've been talking about having "specialized" reviewers, or multi
layered reviewing. There are several things we do in reviewing, and for
big enough patches there's no need to have the same reviewer do all of
them.

[...searching the archives for a proposal I did already send...]
 http://archives.postgresql.org/pgsql-hackers/2009-08/msg00764.php

So this mail proposes we see those separate items to be handled in
review:
- patch (applies, merge, compiles, pass regression)- code reading (looks like it was already there, no WTF?) [1]-
documentation(covers code, targets users, is sufficient)- testing (code behavior is what is documented, works well)-
creativetesting (tried hard to crash it)- perf testing (profiling, no regression in non optimized cases...)- you name
it

Now the senior reviewers you're talking about are required the most for
code reading. We certainly still can have an army of junior reviewers,
or not-wannabe-hackers reviewers checking the other points. That'd push
the bottleneck some.

Regards,
-- 
dim

[1] http://www.osnews.com/images/comics/wtfm.jpg

Robert Haas wrote:
> On Thu, Jan 7, 2010 at 9:11 PM, Andrew Dunstan <andrew@dunslane.net> wrote:
> > This strikes me as quite premature. Heiki just said he expects to have SR
> > committed next week.
> 
> Getting it committed is not what I'm worried about.  What I'm
> concerned about is Tom's statement that he believes that HS is not
> close to being in a state where it is stable enough to go to beta, and
> the possibility that other large patches committed during this
> CommitFest might have similar issues.

Looking at our existing schedule, I think it is unlikely we will even
make a June 2010 release date for 8.5.  Let me explain why --- here is
the ideal schedule:
Jan 15 start commitfestFeb 15 stop commitfestApr  1 start betaJun  1 release release candidate (RC)Jun 20 release 8.5

You can't move from commitfest to beta until all _known_ bugs are
fixed/addressed, and you can't move from beta to RC using the same
criteria.

Of course we rarely have an ideal schedule --- we should expect
unexpected problems.  Now, I know people are going to say I am being
over pessimistic, but history will bear out my analysis.  I think we
should anticipate a July or August release of 8.5.

--  Bruce Momjian  <bruce@momjian.us>        http://momjian.us EnterpriseDB
http://enterprisedb.com
 + If your life is a hard drive, Christ can be your backup. +

Bruce Momjian <bruce@momjian.us> wrote:
> here is the ideal schedule:
> 
>     Jan 15 start commitfest
>     Feb 15 stop commitfest
>     Apr  1 start beta
>     Jun  1 release release candidate (RC)
>     Jun 20 release 8.5
> Of course we rarely have an ideal schedule
So for a project which strives for an annual release, we have over
six months from initial submission of a *small* patch until the
release it goes in, if we meet the ideal schedule?  Longer for large
patches or if we slip from the ideal schedule?  That sounds like
there are still some opportunities for improvements in project
management, but it's not *so* bad if development for the next
release can overlap the tail end of that schedule.  I'm not sure how
much that is anticipated or encouraged, though.
It also seems to call into question the wisdom of annual releases. 
If we had a two-year cycle which had three times as much in it,
would that be an improvement, or not?
-Kevin

"Kevin Grittner" <Kevin.Grittner@wicourts.gov> wrote:
> Bruce Momjian <bruce@momjian.us> wrote:
>>     Jan 15 start commitfest
>>     Jun 20 release 8.5
> over six months
OK, so "over *five* months".  Still....
-Kevin

* Kevin Grittner (Kevin.Grittner@wicourts.gov) wrote:
> It also seems to call into question the wisdom of annual releases.
> If we had a two-year cycle which had three times as much in it,
> would that be an improvement, or not?

At the moment, my vote would be "how 'bout we discuss this post-8.5?".
Maybe if we postpone the discussion till then, we'll actually be able to
chew through everything and release close to on-time, otherwise I wonder
if a thread like this won't end up sucking up too much in terms of our
resources right at crunch time..
    Thanks,
        Stephen

On Jan 8, 2010, at 1:02 AM, Dimitri Fontaine wrote:

> Now, I'll second Greg Smith and Tom here, in that I think we need to run
> the last commitfest as usual, knowing that the outcome of the commitfest
> for any given patch is not "it made it" but "we reviewed it". It's still
> right for the project to bump a patch on resources ground rather than on
> technical merit, at the end of the commitfest.

+1

David

On Fri, Jan 8, 2010 at 11:44 AM, Bruce Momjian <bruce@momjian.us> wrote:
> Robert Haas wrote:
>> On Thu, Jan 7, 2010 at 9:11 PM, Andrew Dunstan <andrew@dunslane.net> wrote:
>> > This strikes me as quite premature. Heiki just said he expects to have SR
>> > committed next week.
>>
>> Getting it committed is not what I'm worried about.  What I'm
>> concerned about is Tom's statement that he believes that HS is not
>> close to being in a state where it is stable enough to go to beta, and
>> the possibility that other large patches committed during this
>> CommitFest might have similar issues.
>
> Looking at our existing schedule, I think it is unlikely we will even
> make a June 2010 release date for 8.5.  Let me explain why --- here is
> the ideal schedule:
>
>        Jan 15 start commitfest
>        Feb 15 stop commitfest
>        Apr  1 start beta
>        Jun  1 release release candidate (RC)
>        Jun 20 release 8.5
>
> You can't move from commitfest to beta until all _known_ bugs are
> fixed/addressed, and you can't move from beta to RC using the same
> criteria.

Hmm.  For 8.4, I don't think we actually fixed "all" known bugs - I
think we made a decision about which ones had to be fixed and which
ones we were going to push off, and then did so.

> Of course we rarely have an ideal schedule --- we should expect
> unexpected problems.  Now, I know people are going to say I am being
> over pessimistic, but history will bear out my analysis.  I think we
> should anticipate a July or August release of 8.5.

I think so, too, but I'm actually afraid that if we don't start making
some tough decisions soon it's going to be even later than that.  I'm
dismayed by the number of people who seem to think that the current
schedule is not already tight.

...Robert

Robert Haas wrote:
> > You can't move from commitfest to beta until all _known_ bugs are
> > fixed/addressed, and you can't move from beta to RC using the same
> > criteria.
> 
> Hmm.  For 8.4, I don't think we actually fixed "all" known bugs - I
> think we made a decision about which ones had to be fixed and which
> ones we were going to push off, and then did so.

Right, we have to decide which ones get pushed to the TODO list, but
also consider that several obvious bugs got into the 8.4.0 release,
which is something we are going to try to avoid this time.

> > Of course we rarely have an ideal schedule --- we should expect
> > unexpected problems. ?Now, I know people are going to say I am being
> > over pessimistic, but history will bear out my analysis. ?I think we
> > should anticipate a July or August release of 8.5.
> 
> I think so, too, but I'm actually afraid that if we don't start making
> some tough decisions soon it's going to be even later than that.  I'm
> dismayed by the number of people who seem to think that the current
> schedule is not already tight.

Perhaps they are now dissuaded.  ;-)

I think once HS and SR are in CVS we could easily be pushed back, but it
seems at least some people are willing to accept that.  The only other
option is to keep them for 8.6 and let them be tested during a full
development cycle.

--  Bruce Momjian  <bruce@momjian.us>        http://momjian.us EnterpriseDB
http://enterprisedb.com
 + If your life is a hard drive, Christ can be your backup. +

On Friday 08 January 2010 19:07:16 Bruce Momjian wrote:
> > I think so, too, but I'm actually afraid that if we don't start making
> > some tough decisions soon it's going to be even later than that.  I'm
> > dismayed by the number of people who seem to think that the current
> > schedule is not already tight.
> 
> Perhaps they are now dissuaded.  ;-)
> 
> I think once HS and SR are in CVS we could easily be pushed back, but it
> seems at least some people are willing to accept that.  The only other
> option is to keep them for 8.6 and let them be tested during a full
> development cycle.
Thats quite similar to where 8.5 started...

Andres

On Fri, Jan 8, 2010 at 1:07 PM, Bruce Momjian <bruce@momjian.us> wrote:
> Robert Haas wrote:
>> > You can't move from commitfest to beta until all _known_ bugs are
>> > fixed/addressed, and you can't move from beta to RC using the same
>> > criteria.
>>
>> Hmm.  For 8.4, I don't think we actually fixed "all" known bugs - I
>> think we made a decision about which ones had to be fixed and which
>> ones we were going to push off, and then did so.
>
> Right, we have to decide which ones get pushed to the TODO list, but
> also consider that several obvious bugs got into the 8.4.0 release,
> which is something we are going to try to avoid this time.

True.  It's worth being clear, though I'm sure we're on the same
wavelength here, that those bugs didn't come from the open items list.They came from stabilization issues related to
largepatches 
committed in that release cycle.  That's why it seems to me that
pushing off some of the large patches that were not submitted until
the final CommitFest is likely to speed up the release.

We discussed doing this at the very beginning of 8.4 release cycle
and, the more I think about it, the more I think it's not fair not to
go ahead and do it.  Otherwise, we're rewarding people for ignoring a
guideline that was discussed, and punishing (1) the people who have
refrained from submitting large patches at the last minute, (2) people
who would like to see their already-committed patches released on a
reasonable time frame, and (3) people who don't want the tree to be
frozen for a near-eternity while we shake out all the bugs that these
large, last-minute patches introduce.  We're also increasing the
chances the the final release will contain undiscovered bugs, since
they will have had ONLY the beta period, and no part of the
development cycle, to shake out.

...Robert

Robert Haas wrote:
> On Fri, Jan 8, 2010 at 1:07 PM, Bruce Momjian <bruce@momjian.us> wrote:
> > Robert Haas wrote:
> >> > You can't move from commitfest to beta until all _known_ bugs are
> >> > fixed/addressed, and you can't move from beta to RC using the same
> >> > criteria.
> >>
> >> Hmm. ?For 8.4, I don't think we actually fixed "all" known bugs - I
> >> think we made a decision about which ones had to be fixed and which
> >> ones we were going to push off, and then did so.
> >
> > Right, we have to decide which ones get pushed to the TODO list, but
> > also consider that several obvious bugs got into the 8.4.0 release,
> > which is something we are going to try to avoid this time.
> 
> True.  It's worth being clear, though I'm sure we're on the same
> wavelength here, that those bugs didn't come from the open items list.
>  They came from stabilization issues related to large patches
> committed in that release cycle.  That's why it seems to me that
> pushing off some of the large patches that were not submitted until
> the final CommitFest is likely to speed up the release.

Yes, these were things that should have showed up in testing, but
didn't.

> We discussed doing this at the very beginning of 8.4 release cycle
> and, the more I think about it, the more I think it's not fair not to
> go ahead and do it.  Otherwise, we're rewarding people for ignoring a
> guideline that was discussed, and punishing (1) the people who have
> refrained from submitting large patches at the last minute, (2) people
> who would like to see their already-committed patches released on a
> reasonable time frame, and (3) people who don't want the tree to be
> frozen for a near-eternity while we shake out all the bugs that these
> large, last-minute patches introduce.  We're also increasing the
> chances the the final release will contain undiscovered bugs, since
> they will have had ONLY the beta period, and no part of the
> development cycle, to shake out.

Doing what?  Not including HS an SR in 8.5?

--  Bruce Momjian  <bruce@momjian.us>        http://momjian.us EnterpriseDB
http://enterprisedb.com
 + If your life is a hard drive, Christ can be your backup. +

On Fri, Jan 8, 2010 at 1:29 PM, Bruce Momjian <bruce@momjian.us> wrote:
> Robert Haas wrote:
>> On Fri, Jan 8, 2010 at 1:07 PM, Bruce Momjian <bruce@momjian.us> wrote:
>> > Robert Haas wrote:
>> >> > You can't move from commitfest to beta until all _known_ bugs are
>> >> > fixed/addressed, and you can't move from beta to RC using the same
>> >> > criteria.
>> >>
>> >> Hmm. ?For 8.4, I don't think we actually fixed "all" known bugs - I
>> >> think we made a decision about which ones had to be fixed and which
>> >> ones we were going to push off, and then did so.
>> >
>> > Right, we have to decide which ones get pushed to the TODO list, but
>> > also consider that several obvious bugs got into the 8.4.0 release,
>> > which is something we are going to try to avoid this time.
>>
>> True.  It's worth being clear, though I'm sure we're on the same
>> wavelength here, that those bugs didn't come from the open items list.
>>  They came from stabilization issues related to large patches
>> committed in that release cycle.  That's why it seems to me that
>> pushing off some of the large patches that were not submitted until
>> the final CommitFest is likely to speed up the release.
>
> Yes, these were things that should have showed up in testing, but
> didn't.
>
>> We discussed doing this at the very beginning of 8.4 release cycle
>> and, the more I think about it, the more I think it's not fair not to
>> go ahead and do it.  Otherwise, we're rewarding people for ignoring a
>> guideline that was discussed, and punishing (1) the people who have
>> refrained from submitting large patches at the last minute, (2) people
>> who would like to see their already-committed patches released on a
>> reasonable time frame, and (3) people who don't want the tree to be
>> frozen for a near-eternity while we shake out all the bugs that these
>> large, last-minute patches introduce.  We're also increasing the
>> chances the the final release will contain undiscovered bugs, since
>> they will have had ONLY the beta period, and no part of the
>> development cycle, to shake out.
>
> Doing what?  Not including HS an SR in 8.5?

No.  Pushing off large patches that weren't submitted until the last
CommitFest to the next release.

...Robert

Robert Haas wrote:
> >> We discussed doing this at the very beginning of 8.4 release cycle
> >> and, the more I think about it, the more I think it's not fair not to
> >> go ahead and do it. ?Otherwise, we're rewarding people for ignoring a
> >> guideline that was discussed, and punishing (1) the people who have
> >> refrained from submitting large patches at the last minute, (2) people
> >> who would like to see their already-committed patches released on a
> >> reasonable time frame, and (3) people who don't want the tree to be
> >> frozen for a near-eternity while we shake out all the bugs that these
> >> large, last-minute patches introduce. ?We're also increasing the
> >> chances the the final release will contain undiscovered bugs, since
> >> they will have had ONLY the beta period, and no part of the
> >> development cycle, to shake out.
> >
> > Doing what? ?Not including HS an SR in 8.5?
> 
> No.  Pushing off large patches that weren't submitted until the last
> CommitFest to the next release.

Sorry, I am still confused.  "Last" is the previous commit-fest,
November, or the final commit-fest, January.  Please restate your
opinion in full.  Thanks.

--  Bruce Momjian  <bruce@momjian.us>        http://momjian.us EnterpriseDB
http://enterprisedb.com
 + If your life is a hard drive, Christ can be your backup. +

On Fri, Jan 8, 2010 at 1:38 PM, Bruce Momjian <bruce@momjian.us> wrote:
> Robert Haas wrote:
>> >> We discussed doing this at the very beginning of 8.4 release cycle
>> >> and, the more I think about it, the more I think it's not fair not to
>> >> go ahead and do it. ?Otherwise, we're rewarding people for ignoring a
>> >> guideline that was discussed, and punishing (1) the people who have
>> >> refrained from submitting large patches at the last minute, (2) people
>> >> who would like to see their already-committed patches released on a
>> >> reasonable time frame, and (3) people who don't want the tree to be
>> >> frozen for a near-eternity while we shake out all the bugs that these
>> >> large, last-minute patches introduce. ?We're also increasing the
>> >> chances the the final release will contain undiscovered bugs, since
>> >> they will have had ONLY the beta period, and no part of the
>> >> development cycle, to shake out.
>> >
>> > Doing what? ?Not including HS an SR in 8.5?
>>
>> No.  Pushing off large patches that weren't submitted until the last
>> CommitFest to the next release.
>
> Sorry, I am still confused.  "Last" is the previous commit-fest,
> November, or the final commit-fest, January.  Please restate your
> opinion in full.  Thanks.

Argh, sorry.

Pushing off large patches that weren't submitted prior to the final
CommitFest of the development cycle, namely, the January CommitFest.

...Robert

Jeff,

> Aside: I'll take this alarm as a very strong hint that I shouldn't push
> the "range types" any more until the next development cycle.
> Particularly because Tom is one of the people with opinions about it, so
> I don't want to distract him from features submitted several commitfests
> ago.

Yeah, as much as I love range types (and will use them) they are
distributable separately from PostgreSQL, and can be used as add-ons
until 8.6.

--Josh Berkus

> Presuming enough reviewers (which should be the case this time given the
> expectation that submitters also review), the suggested pacing here now
> has every patch passing through a round of review and potentially one
> update within ten days.  

If we *don't* have enough reviewers, though, I think Robert is within
his rights as CFM to decide that large patches which are appearing in
this CF for the first time do not get reviewed.  We discussed that
possibility back when we set the schedule, and I think it's still
possible that we'll need to resort to it.

--Josh Berkus

Robert Haas wrote:
> On Fri, Jan 8, 2010 at 1:38 PM, Bruce Momjian <bruce@momjian.us> wrote:
> > Robert Haas wrote:
> >> >> We discussed doing this at the very beginning of 8.4 release cycle
> >> >> and, the more I think about it, the more I think it's not fair not to
> >> >> go ahead and do it. ?Otherwise, we're rewarding people for ignoring a
> >> >> guideline that was discussed, and punishing (1) the people who have
> >> >> refrained from submitting large patches at the last minute, (2) people
> >> >> who would like to see their already-committed patches released on a
> >> >> reasonable time frame, and (3) people who don't want the tree to be
> >> >> frozen for a near-eternity while we shake out all the bugs that these
> >> >> large, last-minute patches introduce. ?We're also increasing the
> >> >> chances the the final release will contain undiscovered bugs, since
> >> >> they will have had ONLY the beta period, and no part of the
> >> >> development cycle, to shake out.
> >> >
> >> > Doing what? ?Not including HS an SR in 8.5?
> >>
> >> No. ?Pushing off large patches that weren't submitted until the last
> >> CommitFest to the next release.
> >
> > Sorry, I am still confused. ?"Last" is the previous commit-fest,
> > November, or the final commit-fest, January. ?Please restate your
> > opinion in full. ?Thanks.
> 
> Argh, sorry.
> 
> Pushing off large patches that weren't submitted prior to the final
> CommitFest of the development cycle, namely, the January CommitFest.

Yea, I thought that was standard policy since we started commit-fests.

--  Bruce Momjian  <bruce@momjian.us>        http://momjian.us EnterpriseDB
http://enterprisedb.com
 + If your life is a hard drive, Christ can be your backup. +

On fre, 2010-01-08 at 10:02 +0100, Dimitri Fontaine wrote:
> Now, I'll second Greg Smith and Tom here, in that I think we need to
> run
> the last commitfest as usual, knowing that the outcome of the
> commitfest
> for any given patch is not "it made it" but "we reviewed it". It's
> still
> right for the project to bump a patch on resources ground rather than
> on
> technical merit, at the end of the commitfest. 

+1, leave everything as is.

The commitfest is a tool for people to track what is going on, not a
tool to tell people what to do.

On Fri, Jan 8, 2010 at 7:48 PM, Peter Eisentraut <peter_e@gmx.net> wrote:
> On fre, 2010-01-08 at 10:02 +0100, Dimitri Fontaine wrote:
>> Now, I'll second Greg Smith and Tom here, in that I think we need to
>> run
>> the last commitfest as usual, knowing that the outcome of the
>> commitfest
>> for any given patch is not "it made it" but "we reviewed it". It's
>> still
>> right for the project to bump a patch on resources ground rather than
>> on
>> technical merit, at the end of the commitfest.
>
> +1, leave everything as is.
>
> The commitfest is a tool for people to track what is going on, not a
> tool to tell people what to do.

I don't understand what you mean by this.  Can you please elaborate?

It's obvious to me that there is a strong consensus against bumping
any patches from the final CommitFest at this point.  Most people seem
to feel that this is premature, and some people seem to find it
outright ludicrous.  I don't really understand why.  I have always
felt that the purpose of a CommitFest was to give everyone a fair
shake at getting their patch reviewed, provided that they followed
certain ground rules.  And I thought we had agreement that one of
those ground rules was "don't submit new, large patches to the final
CommitFest in a particular release cycle".  No?

...Robert

Robert Haas <robertmhaas@gmail.com> writes:
>  I have always
> felt that the purpose of a CommitFest was to give everyone a fair
> shake at getting their patch reviewed, provided that they followed
> certain ground rules.  

Yes, like for example submitting the patch before the commit fest
begins.

> And I thought we had agreement that one of
> those ground rules was "don't submit new, large patches to the final
> CommitFest in a particular release cycle".  No?

I don't remember this having been agreed upon. What I think have been
said before is that doing so would not help stabilizing the tree before
release.

You seem to be wanting to put a lot of energy into being successful at
following the current release schedule, which others seem to be seeing
as an hint or a wish more than anything else (it's the expected one, not
the one we're committed to, I'd venture).

Is it more important to follow the calendar or to be unable to know with
a month precision when we're going to release the best possible 8.5?
Again, it's a compromise to find. You're pushing towards the calendar,
we're advocating staying fair (opened?) with contributors even when it
means we're taking risks on the schedule.

Regards,
-- 
dim

On Sat, Jan 9, 2010 at 8:07 AM, Dimitri Fontaine <dfontaine@hi-media.com> wrote:
> Robert Haas <robertmhaas@gmail.com> writes:
>>  I have always
>> felt that the purpose of a CommitFest was to give everyone a fair
>> shake at getting their patch reviewed, provided that they followed
>> certain ground rules.
>
> Yes, like for example submitting the patch before the commit fest
> begins.

Right.

>> And I thought we had agreement that one of
>> those ground rules was "don't submit new, large patches to the final
>> CommitFest in a particular release cycle".  No?
>
> I don't remember this having been agreed upon.

As far as I know we have always had this rule.  Here is Tom talking
about having it for 8.4 and trying to formalize it for 8.5.

http://archives.postgresql.org/pgsql-hackers/2009-06/msg01520.php

Here is Josh discussing the details of how the rule should be
implemented, while agreeing that it exists:

http://archives.postgresql.org/pgsql-hackers/2009-09/msg00141.php

There are also various messages in the archives where various patch
authors discuss development plans they have made to avoid running
afoul of this rule.

Basically, here's my feeling.  Either we have a rule that we can
bounce large, previously-unseen patches from the final CommitFest of
the release cycle, or we don't.  If we do, then we should go ahead and
do it, and we should do it early when it will have more effect rather
than putting a lot of time into those patches and doing it only once
the release is already late.  On the other hand, if we don't, then we
should have to core team publish a clear statement that all
CommitFests are equal and we're just going to slip the schedule if
there are too many patches for the last one.

Right now, what we have is some patch authors (like Jeff Davis)
holding off from submitting big new features to the last CommitFest,
essentially voluntarily, and others (like KaiGai Kohei, and I think
perhaps also Simon Riggs) making Herculean attempts to meet certain
deadlines even when they seemed somewhat artificial.  Then on the flip
side we have others like Teodor and Oleg who submitted large patches
at the end of the development cycle.  Of course, Teodor and Oleg are
free to do their development whenever they like, but why should we
review their work and not Jeff's?  It seems to me that having a rule
and not enforcing it is the worst of all possible worlds: it
essentially punishes people who have voluntarily followed it.

From a practical standpoint, it seems much better to me to have the
rule than not, because committing large patches earlier in the cycle
seems better from the point of view of having them well-tested and
stabilized before the release comes out.  But if the consensus is that
we have no such rule, then let's be honest about it.

> What I think have been
> said before is that doing so would not help stabilizing the tree before
> release.

Sorry, I'm not following this sentence.

> You seem to be wanting to put a lot of energy into being successful at
> following the current release schedule, which others seem to be seeing
> as an hint or a wish more than anything else (it's the expected one, not
> the one we're committed to, I'd venture).
>
> Is it more important to follow the calendar or to be unable to know with
> a month precision when we're going to release the best possible 8.5?
> Again, it's a compromise to find. You're pushing towards the calendar,
> we're advocating staying fair (opened?) with contributors even when it
> means we're taking risks on the schedule.

I am definitely pushing for the schedule.  It's a maxim of software
development that you can have time-based releases or feature-based
releases, but not both.  In this community, we have time-based
releases early in the cycle and then they change to feature-based
releases late in the cycle.  As we found out with 8.4, trying to be
both on time and feature-complete can result in failing at both.  I
feel that we've been eminently fair to contributors in the 8.5 cycle -
it's something that I have personally worked very hard on - and I
actually also feel that what I am proposing now is also fair.  It may
not be very popular, though.

...Robert

On fre, 2010-01-08 at 21:01 -0500, Robert Haas wrote:
> > The commitfest is a tool for people to track what is going on, not a
> > tool to tell people what to do.
> 
> I don't understand what you mean by this.  Can you please elaborate?

The proposal was apparently that a small, vocal group gets to decide
what features are more important than others, and then change the commit
fest listing to make everyone work on those features instead of some
other ones.  My opinion is that the commit fest should list everything
that was submitted and that participants can decide on their own what is
more important to them.

> And I thought we had agreement that one of
> those ground rules was "don't submit new, large patches to the final
> CommitFest in a particular release cycle".  No?

I don't agree with that or the way it was assessed in this case.

The reason why I make these points is that if you make the commit fest
too selective, then, since you are not the employer of anyone, people
who don't agree with your choices will just ignore them and do something
else.

Robert Haas <robertmhaas@gmail.com> writes:
> Basically, here's my feeling.  Either we have a rule that we can
> bounce large, previously-unseen patches from the final CommitFest of
> the release cycle, or we don't.  If we do, then we should go ahead and
> do it, and we should do it early when it will have more effect rather
> than putting a lot of time into those patches and doing it only once
> the release is already late.  On the other hand, if we don't, then we
> should have to core team publish a clear statement that all
> CommitFests are equal and we're just going to slip the schedule if
> there are too many patches for the last one.

Yeah, problem being we're trying to solve at least two different
problems here: make contributor happier to contribute to PostgreSQL by
giving them early feedback and releasing their code sooner rather than
later, and having a time-based release.

The two points contradicts exactly at the end of the cycle, when we have
to decide we give the priority to the schedule or the feature
set. Knowing that being late now means being even more late on next
cycle, so contributions missing the boat are even more impacted.

All of this is stating the obvious one more time, but I think that's the
reason why the rule is not written on any wall, and why nobody tried to
enforce it in any way yet.

>> What I think have been
>> said before is that doing so would not help stabilizing the tree before
>> release.
>
> Sorry, I'm not following this sentence.

Trying to state some more obvious, so as to be sure we're talking about
the same things.

> I am definitely pushing for the schedule.  It's a maxim of software
> development that you can have time-based releases or feature-based
> releases, but not both.  In this community, we have time-based
> releases early in the cycle and then they change to feature-based
> releases late in the cycle.  As we found out with 8.4, trying to be
> both on time and feature-complete can result in failing at both.  I
> feel that we've been eminently fair to contributors in the 8.5 cycle -
> it's something that I have personally worked very hard on - and I
> actually also feel that what I am proposing now is also fair.  It may
> not be very popular, though.

This has already said before, but let's try it again.

One way to solve the problem could be to have a dedicated release
management team, taking responsibilities after last alpha of the cycle
until release: open items, getting to beta, then fixing any bug testers
find, some advocacy people for having more testers, etc, then release
candidates management and then .0 release.

While this team would work on this, we could maybe have the next cycle
open for development, with its first CommitFest happening while 8.5.0 is
not yet out the door.

Of course it has been said more than once that some resources will have
to be there on both the teams, and that we want people to dedicate to
beta testing rather than new features (which is always more fun).

My guess is that current state of affairs is not working that well,
forcing people to concentrate on stabilizing current beta will push them
to procrastinate if that's not what they want to do. If instead they are
working some more on their next patch, what do we lose?

Now running the release management parallel to the first one or two
commit fest of the next cycle would mean less resources to review and
commit that ones, but maybe a better overall average.

The advantages of doing so, if not clear, are that developments never
stops and it's even easier to return a patch in the last commitfest,
we know when next one begins.

Frankly, forcing people into release management and quality assurance
when they do not want to do it does not sound the best way to offer the
best stable code possible at .0 time. And opening a commitfest were it's
possible nothing will get committed (but only reviewed) because resources
are not available sounds only fair. If you really want your stuff
committed, go help stabilizing the beta.

Regards,
-- 
dim

On Sat, Jan 9, 2010 at 9:33 AM, Peter Eisentraut <peter_e@gmx.net> wrote:
> On fre, 2010-01-08 at 21:01 -0500, Robert Haas wrote:
>> > The commitfest is a tool for people to track what is going on, not a
>> > tool to tell people what to do.
>>
>> I don't understand what you mean by this.  Can you please elaborate?
>
> The proposal was apparently that a small, vocal group gets to decide
> what features are more important than others, and then change the commit
> fest listing to make everyone work on those features instead of some
> other ones.

I'm sorry it came across that way.  That wasn't my intention.  I am
afraid that I may have taken Tom's suggestion more seriously than it
was meant, or more seriously than I should have.  I have been spending
a very large amount of time on PostgreSQL lately for reasons that are
OT for this list, and most of that work has been directed toward
trying to make it possible for us to release on time.  I'm afraid that
I may have let myself get a little too wrapped up in this.  If there
is not a community consensus toward making an on-time release
possible, then it is not a good idea for me to devote as much time as
I have been to helping us get there, because (1) it won't work and (2)
I'll be frustrated when it doesn't.

>> And I thought we had agreement that one of
>> those ground rules was "don't submit new, large patches to the final
>> CommitFest in a particular release cycle".  No?
>
> I don't agree with that

If we accept large patches at the very end of the development cycle,
that's when people will submit them.  You've previously criticized the
high proportion of the release cycle that is set aside for CommitFest
and beta, so I'm surprised to see you advocating for a policy that
seems likely to lengthen the time for which the tree is closed.

> or the way it was assessed in this case.

Specifics?

> The reason why I make these points is that if you make the commit fest
> too selective, then, since you are not the employer of anyone, people
> who don't agree with your choices will just ignore them and do something
> else.

Individual contributors can do whatever they like, and they do.  We
have a number of committers, such as yourself, who could be very
helpful in getting us to release sooner, with more features, or both.
However, so far, Tom Lane is the only committer who has indicated any
willingness or time to work on any of these large patches, and so when
we say we don't want to bump any patches, are we really saying we just
want Tom to fit them into his schedule?  Some people, such as Dimitri,
have opined that we should go ahead and do first-level review of all
of them, and I'm fine with that.  But as far as committer review for
those large patches, we don't seem to have a better plan than hoping
Tom can work it all in.  And he may be able to do that, but he's
clearly said he doesn't think HS is ready for beta, so then beta will
be later.

...Robert

On lör, 2010-01-09 at 14:12 -0500, Robert Haas wrote:
> If we accept large patches at the very end of the development cycle,
> that's when people will submit them.  You've previously criticized the
> high proportion of the release cycle that is set aside for CommitFest
> and beta, so I'm surprised to see you advocating for a policy that
> seems likely to lengthen the time for which the tree is closed.

Just to clarify: I am for sticking to the agreed dates.  If some things
are not ready by the necessary date plus/minus one, they won't make the
release.  If it's obvious earlier that something won't make the date, it
shouldn't be committed, and maybe put on the backburner right now.  But
AFAICT, that's independent of when it was submitted.  Some things that
were submitted just the other day might be almost ready, some things
that were first submitted many months ago are still not ready.

On Sat, Jan 9, 2010 at 4:01 PM, Peter Eisentraut <peter_e@gmx.net> wrote:
> On lör, 2010-01-09 at 14:12 -0500, Robert Haas wrote:
>> If we accept large patches at the very end of the development cycle,
>> that's when people will submit them.  You've previously criticized the
>> high proportion of the release cycle that is set aside for CommitFest
>> and beta, so I'm surprised to see you advocating for a policy that
>> seems likely to lengthen the time for which the tree is closed.
>
> Just to clarify: I am for sticking to the agreed dates.  If some things
> are not ready by the necessary date plus/minus one, they won't make the
> release.  If it's obvious earlier that something won't make the date, it
> shouldn't be committed, and maybe put on the backburner right now.  But
> AFAICT, that's independent of when it was submitted.  Some things that
> were submitted just the other day might be almost ready, some things
> that were first submitted many months ago are still not ready.

The portion of the schedule I'm worried about is the one where we go
to beta by March 7th.

http://archives.postgresql.org/pgsql-hackers/2009-09/msg01251.php

I think we can get all the remaining large patches committed by
February 15th if Tom doesn't start working on the remaining open items
until February 15th - but then I do not think that we will have a beta
on March 7th.

http://archives.postgresql.org/pgsql-hackers/2010-01/msg00663.php

...Robert

Peter,

> Just to clarify: I am for sticking to the agreed dates.  If some things
> are not ready by the necessary date plus/minus one, they won't make the
> release.  If it's obvious earlier that something won't make the date, it
> shouldn't be committed, and maybe put on the backburner right now.  But
> AFAICT, that's independent of when it was submitted.  Some things that
> were submitted just the other day might be almost ready, some things
> that were first submitted many months ago are still not ready.

In that case, Robert's comments about patches to bounce on Day 1 of the
commitfest are still valid, regardless of "patch size".  That is, if
we're getting patches which seem very unlikely to make the cut by
Feburary 15 (like KNNGiST, which currently doesn't even apply), then it
makes sense for the CFM to notify those authors as soon as possible that
their patch is probably last in line to get reviewed.

(btw, I'd have prefered the "no large patches" rule, but it did not get
a firm consensus)

In any case, the CFM has sole authority for allocating the efforts of
reviewers who volunteer for assingment (the RRR).  So the CFM can
certainly assign people to review only on patches they think will make
it, and leave high-risk patches for last.

For example, if I were Robert, I probably wouldn't assign any RRR to
KNNGiST until all patches with a high chance of commit were clear.  Of
course, if the PostGIS community got motivated and put Paul and others
on reviewing it to get it through, then great.  Not every reviewer cares
about all patches equally.

That's completely aside from the concept of "owing" anyone a review.
The last commitfest is really about, "what can we get committed for
8.5?"  This would be the main difference between the last commitfest and
the other 3; during the other commitfests we can afford to devote
resources to reviewing patches just because someone has been a "good
hacker"; in CF4, we really have to be pragmatic about what's going to
make it in, or not.

I'll also say: if we can't make time-based releases work, we're probably
dead as a project.  MySQL and Ingres both tried feature-based releases,
and look where they are now.

--Josh Berkus

On Saturday 09 January 2010 16:32:29 Robert Haas wrote:
> On Sat, Jan 9, 2010 at 4:01 PM, Peter Eisentraut <peter_e@gmx.net> wrote:
> > On lör, 2010-01-09 at 14:12 -0500, Robert Haas wrote:
> >> If we accept large patches at the very end of the development cycle,
> >> that's when people will submit them.  You've previously criticized the
> >> high proportion of the release cycle that is set aside for CommitFest
> >> and beta, so I'm surprised to see you advocating for a policy that
> >> seems likely to lengthen the time for which the tree is closed.
> >
> > Just to clarify: I am for sticking to the agreed dates.  If some things
> > are not ready by the necessary date plus/minus one, they won't make the
> > release.  If it's obvious earlier that something won't make the date, it
> > shouldn't be committed, and maybe put on the backburner right now.  But
> > AFAICT, that's independent of when it was submitted.  Some things that
> > were submitted just the other day might be almost ready, some things
> > that were first submitted many months ago are still not ready.
>
> The portion of the schedule I'm worried about is the one where we go
> to beta by March 7th.
>
> http://archives.postgresql.org/pgsql-hackers/2009-09/msg01251.php
>
> I think we can get all the remaining large patches committed by
> February 15th if Tom doesn't start working on the remaining open items
> until February 15th - but then I do not think that we will have a beta
> on March 7th.
>

1) The goal of any CF should not be that everything submitted gets committed,
but that everything gets reviewed.

2) ISTM what had the most agreement was that the large/ugly patches that show
up late should have no expectation of being committed (though we will try to
review them), and also that they would be first in line for being punted should
we start missing dates.

3) Our biggist problem wrt oscillating between a time based and feature based
release has not been releasing the software, but on never even settling on a
feature set to get into beta with.

Given that, if we follow the normal CF process, by Feb 15 we should have a
clear indication of what is and is not ready for commit, and my suspicion is
that those large patches you are most worried about will have taken care of
themselves (one way or the other)

But I don't see much sense in worrying about it now; the 2 weeks between end
of CF and Beta are when we need to be cut-throat. Given that this time the
"must-have" feature is already in the tree, I think you will find people coming
around quickly to the side of pushing things out rather than fighting to get
things in.

Just my .02

--
Robert Treat
Conjecture: http://www.xzilla.net
Consulting: http://www.omniti.com

Robert Treat <xzilla@users.sourceforge.net> writes:
> ... I don't see much sense in worrying about it now; the 2 weeks between end 
> of CF and Beta are when we need to be cut-throat. Given that this time the 
> "must-have" feature is already in the tree, I think you will find people coming 
> around quickly to the side of pushing things out rather than fighting to get 
> things in.  

I think the other Robert's main point is that getting to beta in only
two weeks is ridiculously optimistic (which I'm afraid I agree with).
I believe that what he's proposing is tossing enough stuff overboard
so that we can finish the January CF in much less than a month, and
thereby have more time for alpha-level testing and stabilization of
the tree.

Now the other approach we could take is that we'll ship *something*
on 7 Mar, even if it's less stable than what we've traditionally
considered to be beta quality.  I don't think that really helps
much though; it just means we need more time in beta.
        regards, tom lane

On Sunday 10 January 2010 01:38:07 Tom Lane wrote:
> Robert Treat <xzilla@users.sourceforge.net> writes:
> > ... I don't see much sense in worrying about it now; the 2 weeks between
> > end of CF and Beta are when we need to be cut-throat. Given that this
> > time the "must-have" feature is already in the tree, I think you will
> > find people coming around quickly to the side of pushing things out
> > rather than fighting to get things in.
>
> I think the other Robert's main point is that getting to beta in only
> two weeks is ridiculously optimistic (which I'm afraid I agree with).
> I believe that what he's proposing is tossing enough stuff overboard
> so that we can finish the January CF in much less than a month, and
> thereby have more time for alpha-level testing and stabilization of
> the tree.
>

I agree with your summary, although I'm not sure it needs to be supported at 
this point. While it hasn't been stated explicitly, I suspect most reviewers 
will be reviewing with the idea of "is there any chance that this is ready for 
commit" in the back of thier heads, and things that aren't will likely get 
pushed off quickly.

> Now the other approach we could take is that we'll ship *something*
> on 7 Mar, even if it's less stable than what we've traditionally
> considered to be beta quality.  I don't think that really helps
> much though; it just means we need more time in beta.
>

There are three reasons I'd probably be comfortable with that; 1) the CF 
process means we've likely had more eyes on the code going in than in past 
releases. 2) the alpha releases mean we should have already had more review 
than in previous releases. 3) so far we're still looking good on pg_migrator, 
which should make it easier for people to test the release once we get into 
beta (which should help speed that cycle up). 

But really if beta slips because we don't like the looks of our open issues 
list, thats signicantly better than the last couple releases where we held 
everything up just to get things into CVS months after feature freeze had 
passed us by. 

-- 
Robert Treat
Conjecture: http://www.xzilla.net
Consulting: http://www.omniti.com

On Sun, Jan 10, 2010 at 05:54, Josh Berkus <josh@agliodbs.com> wrote:
> Peter,
>
>> Just to clarify: I am for sticking to the agreed dates.  If some things
>> are not ready by the necessary date plus/minus one, they won't make the
>> release.  If it's obvious earlier that something won't make the date, it
>> shouldn't be committed, and maybe put on the backburner right now.  But
>> AFAICT, that's independent of when it was submitted.  Some things that
>> were submitted just the other day might be almost ready, some things
>> that were first submitted many months ago are still not ready.
>
> In that case, Robert's comments about patches to bounce on Day 1 of the
> commitfest are still valid, regardless of "patch size".  That is, if
> we're getting patches which seem very unlikely to make the cut by
> Feburary 15 (like KNNGiST, which currently doesn't even apply), then it
> makes sense for the CFM to notify those authors as soon as possible that
> their patch is probably last in line to get reviewed.

If it doesn't even build by the time the CF starts (and didn't just
break in the past couple of days), can it even be considered
submitted? I think it's perfectly fair to bounce something that's been
sitting in "waiting for author" for weeks *before* the CF starts at an
early stage, to focus the resources on things that are up-to-date.


-- Magnus HaganderMe: http://www.hagander.net/Work: http://www.redpill-linpro.com/

On Sun, Jan 10, 2010 at 07:38, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Robert Treat <xzilla@users.sourceforge.net> writes:
>> ... I don't see much sense in worrying about it now; the 2 weeks between end
>> of CF and Beta are when we need to be cut-throat. Given that this time the
>> "must-have" feature is already in the tree, I think you will find people coming
>> around quickly to the side of pushing things out rather than fighting to get
>> things in.
>
> I think the other Robert's main point is that getting to beta in only
> two weeks is ridiculously optimistic (which I'm afraid I agree with).
> I believe that what he's proposing is tossing enough stuff overboard
> so that we can finish the January CF in much less than a month, and
> thereby have more time for alpha-level testing and stabilization of
> the tree.

I realize it's moving the goalposts and not necessraily fair, but we
could always try to cut a week off the CF for some more stabilization
period if we think that'll be enough to make a difference.

> Now the other approach we could take is that we'll ship *something*
> on 7 Mar, even if it's less stable than what we've traditionally
> considered to be beta quality.  I don't think that really helps
> much though; it just means we need more time in beta.

That sounds mor elike a final alpha release, in that case?


-- Magnus HaganderMe: http://www.hagander.net/Work: http://www.redpill-linpro.com/

On Sun, Jan 10, 2010 at 08:09, Robert Treat
<xzilla@users.sourceforge.net> wrote:
> On Sunday 10 January 2010 01:38:07 Tom Lane wrote:
>> Robert Treat <xzilla@users.sourceforge.net> writes:
>> > ... I don't see much sense in worrying about it now; the 2 weeks between
>> > end of CF and Beta are when we need to be cut-throat. Given that this
>> > time the "must-have" feature is already in the tree, I think you will
>> > find people coming around quickly to the side of pushing things out
>> > rather than fighting to get things in.
>>
>> I think the other Robert's main point is that getting to beta in only
>> two weeks is ridiculously optimistic (which I'm afraid I agree with).
>> I believe that what he's proposing is tossing enough stuff overboard
>> so that we can finish the January CF in much less than a month, and
>> thereby have more time for alpha-level testing and stabilization of
>> the tree.
>>
>
> I agree with your summary, although I'm not sure it needs to be supported at
> this point. While it hasn't been stated explicitly, I suspect most reviewers
> will be reviewing with the idea of "is there any chance that this is ready for
> commit" in the back of thier heads, and things that aren't will likely get
> pushed off quickly.

So how about we make that explicitly stated?


>> Now the other approach we could take is that we'll ship *something*
>> on 7 Mar, even if it's less stable than what we've traditionally
>> considered to be beta quality.  I don't think that really helps
>> much though; it just means we need more time in beta.
>>
>
> There are three reasons I'd probably be comfortable with that; 1) the CF
> process means we've likely had more eyes on the code going in than in past
> releases. 2) the alpha releases mean we should have already had more review
> than in previous releases. 3) so far we're still looking good on pg_migrator,
> which should make it easier for people to test the release once we get into
> beta (which should help speed that cycle up).

1 is hopfully right. 2 should be, but we haven't received many bug
reports for people with the alphas. Which certainly *could* mean that
1 made there not be many bugs, but it could also mean that people
aren't really testing it, just tasting it.


> But really if beta slips because we don't like the looks of our open issues
> list, thats signicantly better than the last couple releases where we held
> everything up just to get things into CVS months after feature freeze had
> passed us by.

It is, unless it's because we created those open items by committing
things too late in the cycle, even though it wasn't quite ready, "so
the author won't have to wait another year for a release".

-- Magnus HaganderMe: http://www.hagander.net/Work: http://www.redpill-linpro.com/

On Sat, 2010-01-09 at 08:46 -0500, Robert Haas wrote:

> it seems much better to me to have the rule than not

I think we can overplay the need for lots of rules here and the need to
chase up status every 5 minutes.

The first problem, in previous years, was patches spent too long on the
patch queue. That is now solved, AFAICS, at least to my satisfaction. 

The second problem was spending >4 months on code consolidation at last
commitfest before we go to beta. We are unlikely to avoid that just by
saying "it will be two weeks"; trying to force it will cause arguments,
waste developer time and possibly endanger code quality. AFAICS we are
on track for a much improved situation than before and I'm personally
happy with that also.

Quality > Completeness > Timeliness, IMHO.

-- Simon Riggs           www.2ndQuadrant.com

On Sun, Jan 10, 2010 at 2:09 AM, Robert Treat
<xzilla@users.sourceforge.net> wrote:
> But really if beta slips because we don't like the looks of our open issues
> list, thats signicantly better than the last couple releases where we held
> everything up just to get things into CVS months after feature freeze had
> passed us by.

Yes.  It seems that we're at least not going to let the final
CommitFest drag on and on and on this time, which seems like a
positive step.  But will it help enough to make everyone feel
satisfied with the results?  That's less clear.

...Robert

On Sun, Jan 10, 2010 at 5:31 AM, Magnus Hagander <magnus@hagander.net> wrote:
>> But really if beta slips because we don't like the looks of our open issues
>> list, thats signicantly better than the last couple releases where we held
>> everything up just to get things into CVS months after feature freeze had
>> passed us by.
>
> It is, unless it's because we created those open items by committing
> things too late in the cycle, even though it wasn't quite ready, "so
> the author won't have to wait another year for a release".

Exactly.  Or even - it was ready, but even code that's ready can have
bugs.  The more time you give yourself before release, the more likely
you are to find a few of those.

...Robert

On Sun, Jan 10, 2010 at 5:27 AM, Magnus Hagander <magnus@hagander.net> wrote:
> On Sun, Jan 10, 2010 at 05:54, Josh Berkus <josh@agliodbs.com> wrote:
>> Peter,
>>
>>> Just to clarify: I am for sticking to the agreed dates.  If some things
>>> are not ready by the necessary date plus/minus one, they won't make the
>>> release.  If it's obvious earlier that something won't make the date, it
>>> shouldn't be committed, and maybe put on the backburner right now.  But
>>> AFAICT, that's independent of when it was submitted.  Some things that
>>> were submitted just the other day might be almost ready, some things
>>> that were first submitted many months ago are still not ready.
>>
>> In that case, Robert's comments about patches to bounce on Day 1 of the
>> commitfest are still valid, regardless of "patch size".  That is, if
>> we're getting patches which seem very unlikely to make the cut by
>> Feburary 15 (like KNNGiST, which currently doesn't even apply), then it
>> makes sense for the CFM to notify those authors as soon as possible that
>> their patch is probably last in line to get reviewed.
>
> If it doesn't even build by the time the CF starts (and didn't just
> break in the past couple of days), can it even be considered
> submitted? I think it's perfectly fair to bounce something that's been
> sitting in "waiting for author" for weeks *before* the CF starts at an
> early stage, to focus the resources on things that are up-to-date.

Well, if it's not updated before the CommitFest starts, sure.  But I
suspect an update will come through at the last minute.

A few more large patches may show up, too, and some small ones almost
certainly will.  The number of patches on the CommitFest page
typically increases very quickly in the last few days before we get
started.

...Robert

Josh Berkus wrote:
> I'll also say: if we can't make time-based releases work, we're probably
> dead as a project.  MySQL and Ingres both tried feature-based releases,
> and look where they are now.
>
>
>   

I think you're engaging in a bit of 'post hoc ergo propter hoc' 
reasoning here.

In any case, I think it's a false dichotomy. We always seem to treat 
this as an all or nothing deal. But there is no reason that we must. We 
could easily decide that one or two features were critical for the 
release and everything else would have to make the time cut. For this 
release those features would obviously be HS and SR. Obviously we could 
could at some stage decide to cut our losses and run, as we did last 
release (eventually). But I think that the idea that we should never at 
least have a stated goal to have certain features in a given release is 
a bit silly.

If you think we could put out this coming release without having HS + 
SR, and not do significant damage to the project, I can only say I 
profoundly disagree.

cheers

andrew

On sön, 2010-01-10 at 01:38 -0500, Tom Lane wrote:
> Now the other approach we could take is that we'll ship *something*
> on 7 Mar, even if it's less stable than what we've traditionally
> considered to be beta quality.  I don't think that really helps
> much though; it just means we need more time in beta.

Maybe we need to ponder for a minute what beta ought to mean for us.
With all the new processes and guidelines being established, the
transition criteria into and out of beta are in my mind pretty weakly
defined.

> Now the other approach we could take is that we'll ship *something*
> on 7 Mar, even if it's less stable than what we've traditionally
> considered to be beta quality.  I don't think that really helps
> much though; it just means we need more time in beta.

Well, we're shipping an alpha, aren't we?

My proposal for "beta in 2 weeks" was based on the idea of having *no*
new patches in CF4, at all.  Given the reality of HS+SR, I don't think
it's realistic anymore either.

--Josh Berkus

On Sun, Jan 10, 2010 at 6:24 PM, Josh Berkus <josh@agliodbs.com> wrote:
>> Now the other approach we could take is that we'll ship *something*
>> on 7 Mar, even if it's less stable than what we've traditionally
>> considered to be beta quality.  I don't think that really helps
>> much though; it just means we need more time in beta.
>
> Well, we're shipping an alpha, aren't we?
>
> My proposal for "beta in 2 weeks" was based on the idea of having *no*
> new patches in CF4, at all.  Given the reality of HS+SR, I don't think
> it's realistic anymore either.

Well, the small patches are not going to hold us up much.  I don't
think having to deal with those is going to impact the schedule, even
if they are new.  It's the big ones that are going to drain time from
release prep.

...Robert

Josh Berkus wrote:
> I'll also say: if we can't make time-based releases work, we're probably
> dead as a project.  MySQL and Ingres both tried feature-based releases,
> and look where they are now.

That is a simplification.  We have always done time-based releases with
adjustments for feature/stability.  If we can't factor stability
concerns into the release timetable, we will end up like MySQL.

--  Bruce Momjian  <bruce@momjian.us>        http://momjian.us EnterpriseDB
http://enterprisedb.com
 + If your life is a hard drive, Christ can be your backup. +

On Mon, 2010-01-11 at 19:50 -0500, Bruce Momjian wrote:
> Josh Berkus wrote:
> > I'll also say: if we can't make time-based releases work, we're probably
> > dead as a project.  MySQL and Ingres both tried feature-based releases,
> > and look where they are now.
>
> That is a simplification.  We have always done time-based releases with
> adjustments for feature/stability.  If we can't factor stability
> concerns into the release timetable, we will end up like MySQL.

No, not really.

Joshua D. Drake



--
PostgreSQL.org Major Contributor
Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564
Consulting, Training, Support, Custom Development, Engineering
Respect is earned, not gained through arbitrary and repetitive use or Mr. or Sir.

Robert Treat wrote:
> > Now the other approach we could take is that we'll ship *something*
> > on 7 Mar, even if it's less stable than what we've traditionally
> > considered to be beta quality.  I don't think that really helps
> > much though; it just means we need more time in beta.
> >
> 
> There are three reasons I'd probably be comfortable with that; 1) the CF 
> process means we've likely had more eyes on the code going in than in past 
> releases.

The reality check is that was had commit-fests for 8.4 development and
8.4.0 had more easily-identified bugs than most of our previous .0
releases which didn't use commit-fests.

--  Bruce Momjian  <bruce@momjian.us>        http://momjian.us EnterpriseDB
http://enterprisedb.com
 + If your life is a hard drive, Christ can be your backup. +

Bruce Momjian <bruce@momjian.us> writes:
> Robert Treat wrote:
>> There are three reasons I'd probably be comfortable with that; 1) the CF 
>> process means we've likely had more eyes on the code going in than in past 
>> releases.

> The reality check is that was had commit-fests for 8.4 development and
> 8.4.0 had more easily-identified bugs than most of our previous .0
> releases which didn't use commit-fests.

They weren't "easily identified", or we'd have found them before 8.4.0
release.  I think the notion that 8.4.0 was much worse than previous .0
releases is largely bogus, anyway; we've just forgotten all the bugs in
older releases ...
        regards, tom lane

On Mon, Jan 11, 2010 at 07:50:23PM -0500, Bruce Momjian wrote:
> Josh Berkus wrote:
> > I'll also say: if we can't make time-based releases work, we're
> > probably dead as a project.  MySQL and Ingres both tried
> > feature-based releases, and look where they are now.
> 
> That is a simplification.  We have always done time-based releases
> with adjustments for feature/stability.  If we can't factor
> stability concerns into the release timetable, we will end up like
> MySQL.

Comparisons to MySQL or other proprietary software just aren't
pertinent to this discussion.

If we're looking to other projects, the Linux kernel might be more
relevant.  That has, for the nonce, decided to sacrifice stability for
time-based releases.  People who help package the kernel for
distributions are better qualified than I to discuss the merits of
this approach.

Cheers,
David.
-- 
David Fetter <david@fetter.org> http://fetter.org/
Phone: +1 415 235 3778  AIM: dfetter666  Yahoo!: dfetter
Skype: davidfetter      XMPP: david.fetter@gmail.com
iCal: webcal://www.tripit.com/feed/ical/people/david74/tripit.ics

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

> They weren't "easily identified", or we'd have found them before 8.4.0
> release.  I think the notion that 8.4.0 was much worse than previous .0
> releases is largely bogus, anyway; we've just forgotten all the bugs in
> older releases ...

It was worse than some, and better than others.

Bruce's point that the CFs do not enhance stability, however, is taken.The CFs were not really designed to, for that
matter;the purpose of
 
the CFs was to primarily speed up and improve the patch
submission-and-review process.  This is orthangonal to stability, and in
addition I think the CFs have increased the number of patches committed,
which means that even if we'd reduced the number of bugs per patch, we
could still have more bugs overall.

Wider/earlier community testing on the alphas *would* enhance stability.However, despite publicity there just hasn't
beenmuch uptake on
 
testing the alphas.  I'll see what I can do during the beta period.

--Josh Berkus

On Mon, Jan 11, 2010 at 8:14 PM, David Fetter <david@fetter.org> wrote:
> On Mon, Jan 11, 2010 at 07:50:23PM -0500, Bruce Momjian wrote:
>> Josh Berkus wrote:
>> > I'll also say: if we can't make time-based releases work, we're
>> > probably dead as a project.  MySQL and Ingres both tried
>> > feature-based releases, and look where they are now.
>>
>> That is a simplification.  We have always done time-based releases
>> with adjustments for feature/stability.  If we can't factor
>> stability concerns into the release timetable, we will end up like
>> MySQL.
>
> Comparisons to MySQL or other proprietary software just aren't
> pertinent to this discussion.
>
> If we're looking to other projects, the Linux kernel might be more
> relevant.  That has, for the nonce, decided to sacrifice stability for
> time-based releases.  People who help package the kernel for
> distributions are better qualified than I to discuss the merits of
> this approach.

The decision as to whether to have time-based releases or
feature-based releases is a separate issue from the decision about
whether to have well-tested, thought-to-be-bug free releases or
slapped-together, crufty releases.  I suspect I speak for everyone
here when I say that the stability of our releases is a point of pride
for the project and that we would never consider sacrificing it.

The difference between a time-based release and a feature-based
release is that a time-based release is planned to occur at a certain
time.  A feature-based release is planned to occur when certain
features are completed.  Of course, the disadvantage of time-based
releases is that you can never be sure exactly what they will contain
until you get to the end of the development cycle and see what got
done.  With a feature-based release, you can have more confidence that
the features you were expecting will be in the next release, but you
might have to wait a very long time for that release to actually
happen.  For a project like PostgreSQL, which has none of its own
staff and where anyone can change their mind not only about the timing
of a particular patch but also about doing it at all, you might be
waiting a very long time indeed.

Time-based releases do present a certain challenge in terms of release
management.  If we target a release for June, and we really want to
have that release happen in June, then we have to back up from June
and think about where we need to be in, say, January.  If we find
ourselves behind, we have to be willing to throw over some features to
make our scheduled time.  Of course, this is not an exact science, but
you estimate what you need to do and take your best shot.  Enforcing
rules like "no large, new patches in the final CommitFest" can help us
make reasonable decisions about which things should be postponed.

The consensus view on this thread seems to be that we should have a
time-based code freeze, but not a time-based release.  No one has
argued (and I sincerely hope no one will argue) that we should let the
last CommitFest drag on and on, as we did for 8.4.  However, many
people are still eager to see us commit more large patches even though
they know that there are stop-ship issues in CVS HEAD as a result of
Hot Standby, and despite the fact that committing more large patches
will likely add more such issues.  So, barring the possibility that we
somehow pull a collective rabbit out of our hat, we will NOT be ready
for beta on March 1.  I have not yet given up hope on April 1, but I
wouldn't bet on it, either.

As someone pointed out upthread, even if the only thing we do
differently relative to 8.4 is end the last CommitFest on time (i.e.
time-based code freeze), that's an improvement.  However, my
frustration with 8.4 was not only that the last CommitFest was very
long, but also that it took quite a long time after the last
CommitFest was concluded to get the release out the door.  The result
was that the tree was closed to new patches for a very long time (8
months).  I would like to see us shorten that interval this time
around, and I would like to see us be more aggressive in trying to do
so.  I think the theory that CommitFests and/or alpha releases and/or
pg_migrator are going to speed up the final release over previous
cycles is, unfortunately, wishful thinking: I strongly suspect that
the limiting factor is going to be how quickly can we can get to beta.

Upthread, Peter asked what it means to get to beta.  Someone with more
seniority on the project than me will certainly make the decision on
when beta actually happens, but as I'm using it here, I mean that any
open issues that require significant code changes and/or catversion
bumps will have been addressed.  We should not be aware of any open
issues that require fixes we wouldn't be willing to back-patch if they
were first found after release.  That is the milestone which I do not
believe we are currently on track to meet in a timely fashion.

...Robert

Robert Haas wrote:
> The consensus view on this thread seems to be that we should have a
> time-based code freeze, but not a time-based release.  No one has
> argued (and I sincerely hope no one will argue) that we should let the
> last CommitFest drag on and on, as we did for 8.4.  However, many
> people are still eager to see us commit more large patches even though
> they know that there are stop-ship issues in CVS HEAD as a result of
> Hot Standby, and despite the fact that committing more large patches
> will likely add more such issues.  So, barring the possibility that we
> somehow pull a collective rabbit out of our hat, we will NOT be ready
> for beta on March 1.  I have not yet given up hope on April 1, but I
> wouldn't bet on it, either.

I think the big issue with 8.4 was, do we close the commit-fest when we
have open issues, and we aren't clear on how to fix them?  A lot of
unresolve issues get kept for that pre-beta period because all of a
sudden we have to resolve all those complex problems.  I don't see that
changing for 8.5.

What amazes me is how many people who closely follow our development are
mystified by what we do during that pre-beta period.

--  Bruce Momjian  <bruce@momjian.us>        http://momjian.us EnterpriseDB
http://enterprisedb.com
 + If your life is a hard drive, Christ can be your backup. +

On Mon, Jan 11, 2010 at 9:30 PM, Bruce Momjian <bruce@momjian.us> wrote:
> Robert Haas wrote:
>> The consensus view on this thread seems to be that we should have a
>> time-based code freeze, but not a time-based release.  No one has
>> argued (and I sincerely hope no one will argue) that we should let the
>> last CommitFest drag on and on, as we did for 8.4.  However, many
>> people are still eager to see us commit more large patches even though
>> they know that there are stop-ship issues in CVS HEAD as a result of
>> Hot Standby, and despite the fact that committing more large patches
>> will likely add more such issues.  So, barring the possibility that we
>> somehow pull a collective rabbit out of our hat, we will NOT be ready
>> for beta on March 1.  I have not yet given up hope on April 1, but I
>> wouldn't bet on it, either.
>
> I think the big issue with 8.4 was, do we close the commit-fest when we
> have open issues, and we aren't clear on how to fix them?  A lot of
> unresolve issues get kept for that pre-beta period because all of a
> sudden we have to resolve all those complex problems.  I don't see that
> changing for 8.5.

I agree we have some complex issue to work out.  What I disagree with
is waiting until the last CommitFest is over to start working on them.I think we should be talking about them now and
divvyingthem up. 

> What amazes me is how many people who closely follow our development are
> mystified by what we do during that pre-beta period.

Hey, I'm still mystified.  Maybe you and Tom could do twice-a-week
status updates on what you're working on and anything you could use
help with, or something.

...Robert

Robert Haas wrote:
> On Mon, Jan 11, 2010 at 9:30 PM, Bruce Momjian <bruce@momjian.us> wrote:
> > Robert Haas wrote:
> >> The consensus view on this thread seems to be that we should have a
> >> time-based code freeze, but not a time-based release. ?No one has
> >> argued (and I sincerely hope no one will argue) that we should let the
> >> last CommitFest drag on and on, as we did for 8.4. ?However, many
> >> people are still eager to see us commit more large patches even though
> >> they know that there are stop-ship issues in CVS HEAD as a result of
> >> Hot Standby, and despite the fact that committing more large patches
> >> will likely add more such issues. ?So, barring the possibility that we
> >> somehow pull a collective rabbit out of our hat, we will NOT be ready
> >> for beta on March 1. ?I have not yet given up hope on April 1, but I
> >> wouldn't bet on it, either.
> >
> > I think the big issue with 8.4 was, do we close the commit-fest when we
> > have open issues, and we aren't clear on how to fix them? ?A lot of
> > unresolve issues get kept for that pre-beta period because all of a
> > sudden we have to resolve all those complex problems. ?I don't see that
> > changing for 8.5.
> 
> I agree we have some complex issue to work out.  What I disagree with
> is waiting until the last CommitFest is over to start working on them.
>  I think we should be talking about them now and divvying them up.

We could if we could all stop long enough to address them.  I think
there is the feeling that a great idea will pop up eventually, and only
when we are looking at beta do we realize we are out of time, and the
hard, sometime distasteful/ugly, decisions have to be made.

> > What amazes me is how many people who closely follow our development are
> > mystified by what we do during that pre-beta period.
> 
> Hey, I'm still mystified.  Maybe you and Tom could do twice-a-week
> status updates on what you're working on and anything you could use
> help with, or something.

Yea, that would probably help.  I know I am not very transparent in this
area.

--  Bruce Momjian  <bruce@momjian.us>        http://momjian.us EnterpriseDB
http://enterprisedb.com
 + If your life is a hard drive, Christ can be your backup. +

On Mon, Jan 11, 2010 at 9:45 PM, Bruce Momjian <bruce@momjian.us> wrote:
> We could if we could all stop long enough to address them.  I think
> there is the feeling that a great idea will pop up eventually, and only
> when we are looking at beta do we realize we are out of time, and the
> hard, sometime distasteful/ugly, decisions have to be made.

I think there's definitely some truth to that.  It's basically a
project management issue: how do we make sure that decisions get made
on a way forward in a timely fashion?  Of course, you already know my
opinion: the first thing we need is a good list of what the issues
are.  :-)

>> > What amazes me is how many people who closely follow our development are
>> > mystified by what we do during that pre-beta period.
>>
>> Hey, I'm still mystified.  Maybe you and Tom could do twice-a-week
>> status updates on what you're working on and anything you could use
>> help with, or something.
>
> Yea, that would probably help.  I know I am not very transparent in this
> area.

Just my opinion: I think it would be awesome.

...Robert

Robert Haas wrote:
> On Mon, Jan 11, 2010 at 9:45 PM, Bruce Momjian <bruce@momjian.us> wrote:
> > We could if we could all stop long enough to address them. ?I think
> > there is the feeling that a great idea will pop up eventually, and only
> > when we are looking at beta do we realize we are out of time, and the
> > hard, sometime distasteful/ugly, decisions have to be made.
> 
> I think there's definitely some truth to that.  It's basically a
> project management issue: how do we make sure that decisions get made
> on a way forward in a timely fashion?  Of course, you already know my
> opinion: the first thing we need is a good list of what the issues
> are.  :-)
> 
> >> > What amazes me is how many people who closely follow our development are
> >> > mystified by what we do during that pre-beta period.
> >>
> >> Hey, I'm still mystified. ?Maybe you and Tom could do twice-a-week
> >> status updates on what you're working on and anything you could use
> >> help with, or something.
> >
> > Yea, that would probably help. ?I know I am not very transparent in this
> > area.
> 
> Just my opinion: I think it would be awesome.

I tried publishing my email box but it we heavily criticized.  The
problem is that the status of the items is very fluid and I don't have
much time to update status because I am usually spending time trying to
close them.

--  Bruce Momjian  <bruce@momjian.us>        http://momjian.us EnterpriseDB
http://enterprisedb.com
 + If your life is a hard drive, Christ can be your backup. +

Bruce Momjian wrote:
> I think the big issue with 8.4 was, do we close the commit-fest when we
> have open issues, and we aren't clear on how to fix them?  A lot of
> unresolve issues get kept for that pre-beta period because all of a
> sudden we have to resolve all those complex problems.  I don't see that
> changing for 8.5.
>   
That's not quite how I remember it; let's compare directly.  Right now 
we have this:  http://wiki.postgresql.org/wiki/PostgreSQL_8.5_Open_Items

And at this point in the year 8.4 looked like this:  
http://wiki.postgresql.org/index.php?title=PostgreSQL_8.4_Open_Items&oldid=4121

You wrote a good summary of the open issues for 8.4 at the end of 
January that I think is informative reading for how things are compared 
with then:  
http://archives.postgresql.org/message-id/200901270402.n0R42Xj01525@momjian.us

That point had three major features in it that all got pushed to 8.5, 
and the whole in-place upgrade situation was also completely active as 
well--Bruce finishing up a first good pg_migrator beta on 2009-02-18.  
That one I'd consider a fourth major feature open at that point left off 
that list.

Now, considering that list of four, one is committed this time around 
(Hot standby), another quite clearly tabled already (SEPostgreSQL), the 
third is still active but in much better shape (Streaming Replication), 
and the fourth (in-place upgrade) just closed its main list of open 
issues specific to this release.

How about bugs?  The big data dump of open issues and known bugs that 
Tom put together didn't show up until March 26: 
http://wiki.postgresql.org/index.php?title=PostgreSQL_8.4_Open_Items&direction=prev&oldid=5588
http://archives.postgresql.org/message-id/24552.1238093548@sss.pgh.pa.us

Even with that whole mess, the 8.4 beta started on 2009-04-15, and 8.4.0 
made it out on July 1.

Concerns about trimming the CF list are certainly valid, but I think any 
comparison with 8.4 should recognize that there were four giant 
patches/issues floating around at this point for that release, while 
this time there's only a much closer to release SR.  While the rest of 
the patches Robert is concerned about have their complications and 
aren't trivial, there's not a one of them that's anybody is going to 
fight over the way HS, SEPostgreSQL, and in-place upgrades were.  Not to 
trivialize them or their authors work, but frankly changes to things 
like Listen/Notify and the GIN/GIST changes that seem to be gathering 
heat here seem pretty minor compared to the four giant things that were 
open at this point in the 8.4 cycle--the controversial ones seem like 
they have a pretty low destabilizing potential to me from what I've seen.

Personally, I'd like the topic of a thread on "damage control" to be all 
about testing the one big patch that's already in there (HS), its 
related bits like the VACUUM FULL changes, and potentially SR too.  
Those are things that are touching internals that can introduce all 
sorts of new bugs, in the same way that the FSM changes were pretty 
scary for 8.4.  Those are the things I worry about every day right now, 
not whether, say, a very walled-off indexing change for data that only a 
fraction of the user base uses might destabilize things.  Getting 
excited about pre-emptively kicking out patches that may not even be 
commit candidates anyway is distracting thought from the stuff that 
really matters right now IMHO.  I'm more concerned about how to reach a 
bug list as mature as the one Tom presented at the end of March last 
year earlier in this one, for the features that are already in there.

(Oh, and to head off "well what are you doing about it?", I just updated 
pgbench-tools this week to add some missing 8.4 features, eventually 
working toward supporting some of the new 8.5 stuff before beta too.  
I'm hoping to get a couple of months of machine testing time in between 
now and release time.)

-- 
Greg Smith    2ndQuadrant   Baltimore, MD
PostgreSQL Training, Services and Support
greg@2ndQuadrant.com  www.2ndQuadrant.com

On Mon, Jan 11, 2010 at 11:15 PM, Greg Smith <greg@2ndquadrant.com> wrote:
> Personally, I'd like the topic of a thread on "damage control" to be all
> about testing the one big patch that's already in there (HS), its related
> bits like the VACUUM FULL changes, and potentially SR too.  Those are things
> that are touching internals that can introduce all sorts of new bugs, in the
> same way that the FSM changes were pretty scary for 8.4.  Those are the
> things I worry about every day right now, not whether, say, a very
> walled-off indexing change for data that only a fraction of the user base
> uses might destabilize things.  Getting excited about pre-emptively kicking
> out patches that may not even be commit candidates anyway is distracting
> thought from the stuff that really matters right now IMHO.  I'm more
> concerned about how to reach a bug list as mature as the one Tom presented
> at the end of March last year earlier in this one, for the features that are
> already in there.

I agree.  My main concern in terms of dealing with these outstanding
is that it will distract us, particularly Tom, from stabilizing the
tree, especially HS, VF, and SR.  If the tree were in a releasable
state today I wouldn't be worrying about it.

...Robert

> The consensus view on this thread seems to be that we should have a
> time-based code freeze, but not a time-based release.  No one has
> argued (and I sincerely hope no one will argue) that we should let the
> last CommitFest drag on and on, as we did for 8.4.  However, many
> people are still eager to see us commit more large patches even though
> they know that there are stop-ship issues in CVS HEAD as a result of
> Hot Standby, and despite the fact that committing more large patches
> will likely add more such issues.  So, barring the possibility that we
> somehow pull a collective rabbit out of our hat, we will NOT be ready
> for beta on March 1.  I have not yet given up hope on April 1, but I
> wouldn't bet on it, either.
>

+1

Pavel

On Mon, 2010-01-11 at 19:50 -0500, Bruce Momjian wrote:
> Josh Berkus wrote:
> > I'll also say: if we can't make time-based releases work, we're probably
> > dead as a project.  MySQL and Ingres both tried feature-based releases,
> > and look where they are now.
> 
> That is a simplification.  We have always done time-based releases with
> adjustments for feature/stability.  If we can't factor stability
> concerns into the release timetable, we will end up like MySQL.

No, not really.

Joshua D. Drake



-- 
PostgreSQL.org Major Contributor
Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564
Consulting, Training, Support, Custom Development, Engineering
Respect is earned, not gained through arbitrary and repetitive use or Mr. or Sir.

On Mon, Jan 11, 2010 at 09:19:44PM -0500, Robert Haas wrote:
> I have not yet given up hope on April 1, but I wouldn't bet on it,
> either.

Let's *not* schedule anything for April 1.  There's some history, not
to mention an internet-wide holiday, there.

Cheers,
David.
-- 
David Fetter <david@fetter.org> http://fetter.org/
Phone: +1 415 235 3778  AIM: dfetter666  Yahoo!: dfetter
Skype: davidfetter      XMPP: david.fetter@gmail.com
iCal: webcal://www.tripit.com/feed/ical/people/david74/tripit.ics

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

Robert Haas <robertmhaas@gmail.com> writes:
> I agree.  My main concern in terms of dealing with these outstanding
> is that it will distract us, particularly Tom, from stabilizing the
> tree, especially HS, VF, and SR.  If the tree were in a releasable
> state today I wouldn't be worrying about it.

You sound like you want to drop the last Commit Fest and prepare beta
instead.

Regards,
-- 
dim

On Tue, Jan 12, 2010 at 4:19 AM, Dimitri Fontaine
<dfontaine@hi-media.com> wrote:
> Robert Haas <robertmhaas@gmail.com> writes:
>> I agree.  My main concern in terms of dealing with these outstanding
>> is that it will distract us, particularly Tom, from stabilizing the
>> tree, especially HS, VF, and SR.  If the tree were in a releasable
>> state today I wouldn't be worrying about it.
>
> You sound like you want to drop the last Commit Fest and prepare beta
> instead.

I think I was pretty clear about what I was proposing in the message
with which I started this thread - bump some or all the big,
outstanding patches to leave more time for stabilizing the tree.

Almost everyone said "no".  That's the community's decision and I
accept it, but IMHO it's a tacit decision to slip the release.

...Robert

On Tue, 2010-01-12 at 12:52 -0500, Robert Haas wrote:
> On Tue, Jan 12, 2010 at 4:19 AM, Dimitri Fontaine
> <dfontaine@hi-media.com> wrote:
> > Robert Haas <robertmhaas@gmail.com> writes:
> >> I agree.  My main concern in terms of dealing with these outstanding
> >> is that it will distract us, particularly Tom, from stabilizing the
> >> tree, especially HS, VF, and SR.  If the tree were in a releasable
> >> state today I wouldn't be worrying about it.
> >
> > You sound like you want to drop the last Commit Fest and prepare beta
> > instead.
>
> I think I was pretty clear about what I was proposing in the message
> with which I started this thread - bump some or all the big,
> outstanding patches to leave more time for stabilizing the tree.
>
> Almost everyone said "no".  That's the community's decision and I
> accept it, but IMHO it's a tacit decision to slip the release.

Agreed. (although I think we should bump the big outstanding patches)

Joshua D. Drake

>
> ...Robert
>


--
PostgreSQL.org Major Contributor
Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564
Consulting, Training, Support, Custom Development, Engineering
Respect is earned, not gained through arbitrary and repetitive use or Mr. or Sir.

Robert Haas wrote:
> On Tue, Jan 12, 2010 at 4:19 AM, Dimitri Fontaine
> <dfontaine@hi-media.com> wrote:
> > Robert Haas <robertmhaas@gmail.com> writes:
> >> I agree. ?My main concern in terms of dealing with these outstanding
> >> is that it will distract us, particularly Tom, from stabilizing the
> >> tree, especially HS, VF, and SR. ?If the tree were in a releasable
> >> state today I wouldn't be worrying about it.
> >
> > You sound like you want to drop the last Commit Fest and prepare beta
> > instead.
> 
> I think I was pretty clear about what I was proposing in the message
> with which I started this thread - bump some or all the big,
> outstanding patches to leave more time for stabilizing the tree.
> 
> Almost everyone said "no".  That's the community's decision and I
> accept it, but IMHO it's a tacit decision to slip the release.

Agreed that "it's a tacit decision to slip the release".  If HS and SR
had been tested and stable in October we might have had a chance for
June, but at this point it seems very unlikely.

--  Bruce Momjian  <bruce@momjian.us>        http://momjian.us EnterpriseDB
http://enterprisedb.com
 + If your life is a hard drive, Christ can be your backup. +

"Joshua D. Drake" <jd@commandprompt.com> wrote:
> On Tue, 2010-01-12 at 12:52 -0500, Robert Haas wrote:
>> I think I was pretty clear about what I was proposing in the
>> message with which I started this thread - bump some or all the
>> big, outstanding patches to leave more time for stabilizing the
>> tree.
>> 
>> Almost everyone said "no".  That's the community's decision and I
>> accept it, but IMHO it's a tacit decision to slip the release.
> 
> Agreed. (although I think we should bump the big outstanding
> patches)
I'm also inclined to think that it would be wiser not to try to
include large patches first submitted to the last commitfest, at
least if they pose any significant risk of destabilizing the code or
pulling people qualified to review HS or SR off of those areas
before release.
-Kevin

Robert Haas <robertmhaas@gmail.com> writes:
> On Tue, Jan 12, 2010 at 4:19 AM, Dimitri Fontaine
> <dfontaine@hi-media.com> wrote:
>> You sound like you want to drop the last Commit Fest and prepare beta
>> instead.

> I think I was pretty clear about what I was proposing in the message
> with which I started this thread - bump some or all the big,
> outstanding patches to leave more time for stabilizing the tree.

> Almost everyone said "no".  That's the community's decision and I
> accept it, but IMHO it's a tacit decision to slip the release.

I don't think that was the conclusion.  What I thought we were saying
was that we didn't want to bounce those patches in advance of any CF
review at all.  But IMO we should put the larger patches on a very short
leash: if they don't appear pretty clean and trouble-free, they should
get postponed.  We need to minimize the time spent on new patches, but
we don't have to drive it to zero.
        regards, tom lane

On Tue, 2010-01-12 at 12:52 -0500, Robert Haas wrote:
> On Tue, Jan 12, 2010 at 4:19 AM, Dimitri Fontaine
> <dfontaine@hi-media.com> wrote:
> > Robert Haas <robertmhaas@gmail.com> writes:
> >> I agree.  My main concern in terms of dealing with these outstanding
> >> is that it will distract us, particularly Tom, from stabilizing the
> >> tree, especially HS, VF, and SR.  If the tree were in a releasable
> >> state today I wouldn't be worrying about it.
> >
> > You sound like you want to drop the last Commit Fest and prepare beta
> > instead.
> 
> I think I was pretty clear about what I was proposing in the message
> with which I started this thread - bump some or all the big,
> outstanding patches to leave more time for stabilizing the tree.
> 
> Almost everyone said "no".  That's the community's decision and I
> accept it, but IMHO it's a tacit decision to slip the release.

Agreed. (although I think we should bump the big outstanding patches)

Joshua D. Drake

> 
> ...Robert
> 


-- 
PostgreSQL.org Major Contributor
Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564
Consulting, Training, Support, Custom Development, Engineering
Respect is earned, not gained through arbitrary and repetitive use or Mr. or Sir.

On Thu, Jan 7, 2010 at 10:20 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Robert Haas <robertmhaas@gmail.com> writes:
>> OK, we have a proposal on the table to bump some patches from this
>> CommitFest to free up more committer resources, particularly Tom, to
>> work on Hot Standby and Streaming Replication and attempt to
>> accelerate the process of getting 8.5 out the door.  This proposal
>> needs input from the community.  The affected patches are:
>
>> - Listen/Notify Rewrite.
>> - Writeable CTEs.
>> - more frame options for window functions
>> - knngist
>> - rbtree
>
> Looking at this list again, it strikes me that the listen/notify rewrite
> might need to go in so that we have a sane framework for listen/notify
> with HS.  Treating pg_listener as a replicatable table isn't sane at
> all, whereas with a message-driven notify mechanism we have at least got
> the possibility of shipping the messages to the standby (via WAL) and
> having listeners there.  I don't want to say we'd actually implement
> that in 8.5, but shipping pg_listener tuple updates is just completely
> nuts.
>
> The other four things have no apparent connection to HS/SR so I think
> they could be punted without creating technical issues.  Whether this
> is really necessary from a schedule viewpoint is not clear yet.
>
> My thought about it would be to put these four on the back burner;
> not necessarily bounce them right away, but not put any effort into
> them until we have dealt with the other stuff in the January CF.
> At that point we should have a feel for where we are schedule-wise,
> and in particular we'll know whether SR is in or not ;-)

I think it might be time to revisit this issue.  SR is in, and we have
a week left in the CF, and we have all of the above patches plus 5
small ones left to deal with.  rbtree is close to being committable, I
think; knngist has not been reviewed yet; you (Tom) have claimed the
frame options patch but I haven't seen any update on it in a while; I
doubt either of the other two are ready to commit but I'm not sure how
far they have to go.

Thoughts?

...Robert

Robert,

> I think it might be time to revisit this issue.  SR is in, and we have
> a week left in the CF, and we have all of the above patches plus 5
> small ones left to deal with.  rbtree is close to being committable, I
> think; knngist has not been reviewed yet; you (Tom) have claimed the
> frame options patch but I haven't seen any update on it in a while; I
> doubt either of the other two are ready to commit but I'm not sure how
> far they have to go.

I think, as previously discussed, that we should bounce knngist.    It's
a complex patch and nobody saw anything of it until Jan 15, so I don't
feel bad about it.  Mark Cave-Ayland was going to review it, but
apparently felt that rbtree was the higher priority.

Also, this one:
Provide rowcount for utility SELECTs
... based on your last review does not look anywhere near ready.

We know the following because of endless discussion on -hackers; what
these patches seem to be suffering from is endless arguments over
relatively minor points, it just requires someone to make a decision on
implementation method:
Add on_trusted_init and on_untrusted_init to plperl
Faster CREATE DATABASE by delaying fsync

For the rest, can we just have reviewer reports on readiness?
Writeable CTEs
Package namespace and Safe init cleanup for plperl
More frame options in window functions
Fix large object support in pg_dump

Actually, looking at that list, I think we're in pretty darned good
shape.  That's a pretty small list of things left to commit.  Keep in
mind that last year at this time (week 3 of CF-last) we still had over a
dozen patches completely unreviewed!

--Josh Berkus

Robert Haas <robertmhaas@gmail.com> writes:
>>> ... The affected patches are:
>>> - Listen/Notify Rewrite.
>>> - Writeable CTEs.
>>> - more frame options for window functions
>>> - knngist
>>> - rbtree

> I think it might be time to revisit this issue.  SR is in, and we have
> a week left in the CF, and we have all of the above patches plus 5
> small ones left to deal with.  rbtree is close to being committable, I
> think; knngist has not been reviewed yet; you (Tom) have claimed the
> frame options patch but I haven't seen any update on it in a while; I
> doubt either of the other two are ready to commit but I'm not sure how
> far they have to go.

Yeah, I claimed the window frame patch and then got distracted on the
vacuum full vs HS issue.

Current status on that: I expect to be able to commit the core
relation-mapping patch tomorrow (if the power stays on here, which is
no sure thing given the weather).  There is something like another day
or two's worth of work involved to rip out VACUUM FULL INPLACE.  I
suppose that part could get put off till later, but I'm the kind of
person who prefers to finish a project once it's started.  I feel it
has to get done before we release the final alpha in any case.

Given that we have a week still to go in the CF, I feel fairly
confident of still getting the window frame patch in on time
(assuming that there are indeed no major problems with it).
I have not let go of it for that reason, and also because I doubt
anybody else is qualified to commit it --- AFAIR only Hitoshi-san
and I were really neck-deep in the original window patch.

However, with the deadline fast approaching, I don't feel that I
can also promise to handle writeable CTEs, which is another one
that I'd really like to be the committer for.  Maybe we had better
make a management decision about which of those two is higher
priority to get into 9.0.  Also: I haven't been following either
one terribly closely lately.  If there's reason to think that one is
more likely to be committable than the other, that ought to get
factored into the choice of which to go to first; but I'm not
sure whether that's the case.
        regards, tom lane

On Sat, Feb 06, 2010 at 10:38:00PM -0800, Josh Berkus wrote:
> 
> Add on_trusted_init and on_untrusted_init to plperl

> Package namespace and Safe init cleanup for plperl

Alex Hunsaker has marked the latest version of both of those
as Ready for Committer.

Tim.

On Sun, Feb 7, 2010 at 2:05 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Given that we have a week still to go in the CF, I feel fairly
> confident of still getting the window frame patch in on time
> (assuming that there are indeed no major problems with it).
> I have not let go of it for that reason, and also because I doubt
> anybody else is qualified to commit it --- AFAIR only Hitoshi-san
> and I were really neck-deep in the original window patch.
>
> However, with the deadline fast approaching, I don't feel that I
> can also promise to handle writeable CTEs, which is another one
> that I'd really like to be the committer for.  Maybe we had better
> make a management decision about which of those two is higher
> priority to get into 9.0.  Also: I haven't been following either
> one terribly closely lately.  If there's reason to think that one is
> more likely to be committable than the other, that ought to get
> factored into the choice of which to go to first; but I'm not
> sure whether that's the case.

As between the two, I get the feeling that there is more interest in
writeable CTEs.  But that impression might be wrong, since it's an
unscientific recollection of discussions on -hackers; which are
themselves not representative of anything.

I have not looked at the window functions patch at all, and I haven't
looked at the latest version of writeable CTEs, either.  I will try to
spend some time on it in the next couple of days.  My feeling about
the last version is that it lacked a lot in the documentation
department, and also in the comments department.  Since I don't know
that code very well, that made it hard for me to assess technical
correctness.

...Robert

On Sun, Feb 7, 2010 at 1:38 AM, Josh Berkus <josh@agliodbs.com> wrote:
>> I think it might be time to revisit this issue.  SR is in, and we have
>> a week left in the CF, and we have all of the above patches plus 5
>> small ones left to deal with.  rbtree is close to being committable, I
>> think; knngist has not been reviewed yet; you (Tom) have claimed the
>> frame options patch but I haven't seen any update on it in a while; I
>> doubt either of the other two are ready to commit but I'm not sure how
>> far they have to go.
>
> I think, as previously discussed, that we should bounce knngist.    It's
> a complex patch and nobody saw anything of it until Jan 15, so I don't
> feel bad about it.  Mark Cave-Ayland was going to review it, but
> apparently felt that rbtree was the higher priority.

rbtree is a prerequisite for knngist.  point_ops was too.  I think
we're going to get rbtree done yet, but the main patch seems out of
reach.  There's no way it's going to go in without both pre-commit and
post-commit changes, and I don't think we have time for that now.

> Also, this one:
> Provide rowcount for utility SELECTs
> ... based on your last review does not look anywhere near ready.

I wouldn't worry about that one.  It's a small patch.  It's not going
to suck a lot of anyone's time; it'll either make it, or not.

> We know the following because of endless discussion on -hackers; what
> these patches seem to be suffering from is endless arguments over
> relatively minor points, it just requires someone to make a decision on
> implementation method:
> Add on_trusted_init and on_untrusted_init to plperl

I think this one is in pretty good shape.  I think the ball is Andrew
Dunstan's court to commit it, or not.

> Faster CREATE DATABASE by delaying fsync

Too much attempt to design an abstraction layer for things we can't
abstract; not enough doing something that is good enough for now.
Let's just put in something that's not particularly abstract and we
can rearrange later.

> For the rest, can we just have reviewer reports on readiness?
> Writeable CTEs

This was marked ready by the reviewer a long time ago; but that was
considerably over-optimistic, as the recent comments on that thread
attest.  A major feature without documentation is by definition not
ready.

> Package namespace and Safe init cleanup for plperl

I think this is close. Another one for Andrew Dunstan to make the
call, I believe.

> More frame options in window functions

Not sure.

> Fix large object support in pg_dump

Doesn't matter because this one is an open item.  I'm hoping Itagaki
Takahiro is going to take the lead on this one because he committed
the prior patch that created the situation we're now trying to fix.

> Actually, looking at that list, I think we're in pretty darned good
> shape.  That's a pretty small list of things left to commit.  Keep in
> mind that last year at this time (week 3 of CF-last) we still had over a
> dozen patches completely unreviewed!

Unfortunately, we haven't been very successful this time in attracting
non-committer reviewers.  Only about 20% of the committed patches are
listed as having been reviewed by a non-committer.  Still, I agree
with your overall conclusion.

...Robert

2010/2/7 Oleg Bartunov <oleg@sai.msu.su>:
> On Sun, 7 Feb 2010, Robert Haas wrote:
>
>> On Sun, Feb 7, 2010 at 1:38 AM, Josh Berkus <josh@agliodbs.com> wrote:
>>>>
>>>> I think it might be time to revisit this issue.  SR is in, and we have
>>>> a week left in the CF, and we have all of the above patches plus 5
>>>> small ones left to deal with.  rbtree is close to being committable, I
>>>> think; knngist has not been reviewed yet; you (Tom) have claimed the
>>>> frame options patch but I haven't seen any update on it in a while; I
>>>> doubt either of the other two are ready to commit but I'm not sure how
>>>> far they have to go.
>>>
>>> I think, as previously discussed, that we should bounce knngist.    It's
>>> a complex patch and nobody saw anything of it until Jan 15, so I don't
>>> feel bad about it.  Mark Cave-Ayland was going to review it, but
>>> apparently felt that rbtree was the higher priority.
>
> Hey, I'm lost here, when we previously discussed, that knngist should be
> rejected ?

Huh?  Have you been reading -hackers for the last month?  I first
raised this issue on December 30th, and there has been lots more
discussion of it since then.

http://archives.postgresql.org/pgsql-hackers/2009-12/msg02329.php

> knngist is a legal patch, submitted in time (and discussed in
> -hackers) and it's not our fault, people are busy doing other reviews.

I never said anything about fault.  If there's not enough time to get
something committed, then there isn't. That's not a punishment; it's
just something that sometimes happens to patches submitted near the
end of the cycle.  We've been openly discussing this problem on
-hackers for weeks.  But since you brought it up, let's discuss what
has happened so far and the likelihood that this patch is going to be
committable in the next week.

> Knngist has some prerequisites,  rbtree, for example, and it took a while,
> but now, when we're close to commit rbtree, people can review knngist.

This patch is a group of three related patches: point_ops, rbtree, knngist.

point_ops, the simplest, was initially submitted on November 23rd.  An
updated version was submitted on December 30th.  I reviewed it on
December 31st and made some minor suggestions for improvement, which
Teodor accepted.  It was committed on January 14th - so IOW, 1 review
and 14 days from first review to commit.

rbtree, which was more complex, was also submitted on November 23rd.
I took a quick look on it on December 31st, a more complete review on
January 10th, and a still more complete review on January 20th.  I
reviewed it again on January 25th and again on February 5th; and Mark
Cave-Ayland reviewed it on January 29th.  However, the questions that
I asked yesterday and the suggestions I made for reworking it have yet
to be acted on, so it's going to take at least one more round of
reviewing before this is ready for commit.  Discounting my quick look
on December 31st as not being a real review, that means this patch
will have had at least six rounds of review before commit over about 4
weeks.

knngist is the final and most complex patch.  We have 7 or 8 days left
in the CommitFest.  It has had zero reviews thus far.  Are we going to
accomplish six rounds of review in those 7 or 8 days?  Or maybe more,
since the patch is more complex and has far more interaction with the
rest of the code than rbtree?  I don't find that very realistic.  I
think the only way this is going to get committed in the next week is
if we basically assume that everything is OK and commit it without a
careful review, and I am not in favor of that.  But perhaps someone
else will advocate for it.

Frankly, the politics of the end of the release cycle are a bit
frustrating to me.  If these patches had been submitted a few weeks
sooner, they would have been reviewed in the 2009-11 CommitFest and we
would be in much better shape right now.

...Robert

Robert,

I understand your complaints. I think, the real problem is that some of us
live in the part of word with long holidays in December, while we in Russia
have very long holidays in January. So, about a month we couldn't synchronize
developers and reviewers.  I'm not sure if we took this into account.

In regard to the knngist patch I want to claim, that I and Teodor are here
and willing to answer any questions.

Oleg

PS.

I changed subject not to interfere with other topics.

On Sun, 7 Feb 2010, Robert Haas wrote:

> 2010/2/7 Oleg Bartunov <oleg@sai.msu.su>:
>> On Sun, 7 Feb 2010, Robert Haas wrote:
>>
>>> On Sun, Feb 7, 2010 at 1:38 AM, Josh Berkus <josh@agliodbs.com> wrote:
>>>>>
>>>>> I think it might be time to revisit this issue.  SR is in, and we have
>>>>> a week left in the CF, and we have all of the above patches plus 5
>>>>> small ones left to deal with.  rbtree is close to being committable, I
>>>>> think; knngist has not been reviewed yet; you (Tom) have claimed the
>>>>> frame options patch but I haven't seen any update on it in a while; I
>>>>> doubt either of the other two are ready to commit but I'm not sure how
>>>>> far they have to go.
>>>>
>>>> I think, as previously discussed, that we should bounce knngist.    It's
>>>> a complex patch and nobody saw anything of it until Jan 15, so I don't
>>>> feel bad about it.  Mark Cave-Ayland was going to review it, but
>>>> apparently felt that rbtree was the higher priority.
>>
>> Hey, I'm lost here, when we previously discussed, that knngist should be
>> rejected ?
>
> Huh?  Have you been reading -hackers for the last month?  I first
> raised this issue on December 30th, and there has been lots more
> discussion of it since then.
>
> http://archives.postgresql.org/pgsql-hackers/2009-12/msg02329.php
>
>> knngist is a legal patch, submitted in time (and discussed in
>> -hackers) and it's not our fault, people are busy doing other reviews.
>
> I never said anything about fault.  If there's not enough time to get
> something committed, then there isn't. That's not a punishment; it's
> just something that sometimes happens to patches submitted near the
> end of the cycle.  We've been openly discussing this problem on
> -hackers for weeks.  But since you brought it up, let's discuss what
> has happened so far and the likelihood that this patch is going to be
> committable in the next week.
>
>> Knngist has some prerequisites,  rbtree, for example, and it took a while,
>> but now, when we're close to commit rbtree, people can review knngist.
>
> This patch is a group of three related patches: point_ops, rbtree, knngist.
>
> point_ops, the simplest, was initially submitted on November 23rd.  An
> updated version was submitted on December 30th.  I reviewed it on
> December 31st and made some minor suggestions for improvement, which
> Teodor accepted.  It was committed on January 14th - so IOW, 1 review
> and 14 days from first review to commit.
>
> rbtree, which was more complex, was also submitted on November 23rd.
> I took a quick look on it on December 31st, a more complete review on
> January 10th, and a still more complete review on January 20th.  I
> reviewed it again on January 25th and again on February 5th; and Mark
> Cave-Ayland reviewed it on January 29th.  However, the questions that
> I asked yesterday and the suggestions I made for reworking it have yet
> to be acted on, so it's going to take at least one more round of
> reviewing before this is ready for commit.  Discounting my quick look
> on December 31st as not being a real review, that means this patch
> will have had at least six rounds of review before commit over about 4
> weeks.
>
> knngist is the final and most complex patch.  We have 7 or 8 days left
> in the CommitFest.  It has had zero reviews thus far.  Are we going to
> accomplish six rounds of review in those 7 or 8 days?  Or maybe more,
> since the patch is more complex and has far more interaction with the
> rest of the code than rbtree?  I don't find that very realistic.  I
> think the only way this is going to get committed in the next week is
> if we basically assume that everything is OK and commit it without a
> careful review, and I am not in favor of that.  But perhaps someone
> else will advocate for it.
>
> Frankly, the politics of the end of the release cycle are a bit
> frustrating to me.  If these patches had been submitted a few weeks
> sooner, they would have been reviewed in the 2009-11 CommitFest and we
> would be in much better shape right now.
>
> ...Robert
>
    Regards,        Oleg
_____________________________________________________________
Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru),
Sternberg Astronomical Institute, Moscow University, Russia
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(495)939-16-83, +007(495)939-23-83

Robert,

> As between the two, I get the feeling that there is more interest in
> writeable CTEs.  But that impression might be wrong, since it's an
> unscientific recollection of discussions on -hackers; which are
> themselves not representative of anything.

Writeable CTE is definitely the bigger feature.  Effectively, it allows
people to do in a single query data-transformation operations which
would have taken a stored procedure before.  Think of it as comparable
to the introduction of callbacks in Perl for coolness.

> I have not looked at the window functions patch at all, and I haven't
> looked at the latest version of writeable CTEs, either.  I will try to
> spend some time on it in the next couple of days.  My feeling about
> the last version is that it lacked a lot in the documentation
> department, and also in the comments department.  Since I don't know
> that code very well, that made it hard for me to assess technical
> correctness.

Hmmm, that's potentially lethal.  David Fetter has been doing a lot of
presentations on the feature; surely he could turn them into some
documentation?  David?

--Josh Berkus

On 2010-02-07 22:37 +0200, Josh Berkus wrote:
> Robert,
>> I have not looked at the window functions patch at all, and I haven't
>> looked at the latest version of writeable CTEs, either.  I will try to
>> spend some time on it in the next couple of days.  My feeling about
>> the last version is that it lacked a lot in the documentation
>> department, and also in the comments department.  Since I don't know
>> that code very well, that made it hard for me to assess technical
>> correctness.
> 
> Hmmm, that's potentially lethal.  David Fetter has been doing a lot of
> presentations on the feature; surely he could turn them into some
> documentation?  David?

The documentation has definitely improved from the last time Robert
looked at it, but I fear it still needs some more work.  I'm willing to
do that work, but I need something concrete.


Regards,
Marko Tiikkaja

Marko Tiikkaja <marko.tiikkaja@cs.helsinki.fi> writes:
> The documentation has definitely improved from the last time Robert
> looked at it, but I fear it still needs some more work.  I'm willing to
> do that work, but I need something concrete.

It seems to me documentation is required to get into the source tree
before beta, and as we see with some other patches it's definitely the
case even with our newer procedures that some code gets in without its
documentation properly finished. I guess this amounts to the commiter
willing to fill up the docs later on.

But here it's even better as we have the author willing to stay there
and write needed documentation as soon as community agrees on what that
is.

In case I'm not clear, what I'm saying is that I think we can consider
the writable CTE patch ready for commit even though we still have to
decide what its impacts on documentation should be.

There has to be a difference between last alpha and first beta, right?

Regards,
-- 
dim

On Sun, Feb 7, 2010 at 4:03 PM, Dimitri Fontaine <dfontaine@hi-media.com> wrote:
> Marko Tiikkaja <marko.tiikkaja@cs.helsinki.fi> writes:
>> The documentation has definitely improved from the last time Robert
>> looked at it, but I fear it still needs some more work.  I'm willing to
>> do that work, but I need something concrete.
>
> It seems to me documentation is required to get into the source tree
> before beta, and as we see with some other patches it's definitely the
> case even with our newer procedures that some code gets in without its
> documentation properly finished. I guess this amounts to the commiter
> willing to fill up the docs later on.
>
> But here it's even better as we have the author willing to stay there
> and write needed documentation as soon as community agrees on what that
> is.
>
> In case I'm not clear, what I'm saying is that I think we can consider
> the writable CTE patch ready for commit even though we still have to
> decide what its impacts on documentation should be.

Whether a patch is ready to commit will be up to the committer, and I
am doubtful that anyone other than Tom is qualified to do this one.
Others may feel differently, of course.  The rest of us should focus
our effort on improving the patch, rather than arguing about whether
we would commit it as is.

...Robert

Robert Haas <robertmhaas@gmail.com> writes:
> On Sun, Feb 7, 2010 at 4:03 PM, Dimitri Fontaine <dfontaine@hi-media.com> wrote:
>> In case I'm not clear, what I'm saying is that I think we can consider
>> the writable CTE patch ready for commit even though we still have to
>> decide what its impacts on documentation should be.
>
> Whether a patch is ready to commit will be up to the committer

"Ready for Committer" is what I though but failed to type.

Regards,
-- 
dim

On Sun, 7 Feb 2010, Robert Haas wrote:

> On Sun, Feb 7, 2010 at 1:38 AM, Josh Berkus <josh@agliodbs.com> wrote:
>>> I think it might be time to revisit this issue.  SR is in, and we have
>>> a week left in the CF, and we have all of the above patches plus 5
>>> small ones left to deal with.  rbtree is close to being committable, I
>>> think; knngist has not been reviewed yet; you (Tom) have claimed the
>>> frame options patch but I haven't seen any update on it in a while; I
>>> doubt either of the other two are ready to commit but I'm not sure how
>>> far they have to go.
>>
>> I think, as previously discussed, that we should bounce knngist.    It's
>> a complex patch and nobody saw anything of it until Jan 15, so I don't
>> feel bad about it.  Mark Cave-Ayland was going to review it, but
>> apparently felt that rbtree was the higher priority.

Hey, I'm lost here, when we previously discussed, that knngist should be
rejected ? knngist is a legal patch, submitted in time (and discussed in
-hackers) and it's not our fault, people are busy doing other reviews.
Knngist has some prerequisites,  rbtree, for example, and it took a while,
but now, when we're close to commit rbtree, people can review knngist.

>
> rbtree is a prerequisite for knngist.  point_ops was too.  I think
> we're going to get rbtree done yet, but the main patch seems out of
> reach.  There's no way it's going to go in without both pre-commit and
> post-commit changes, and I don't think we have time for that now.
>

We have a week to get review and do possible required pre-commit changes.
Mark, you can download test data for knngist from
http://www.sai.msu.su/~megera/wiki/2009-11-25
    Regards,        Oleg
_____________________________________________________________
Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru),
Sternberg Astronomical Institute, Moscow University, Russia
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(495)939-16-83, +007(495)939-23-83

On Sun, Feb 7, 2010 at 4:54 PM, Dimitri Fontaine <dfontaine@hi-media.com> wrote:
> Robert Haas <robertmhaas@gmail.com> writes:
>> On Sun, Feb 7, 2010 at 4:03 PM, Dimitri Fontaine <dfontaine@hi-media.com> wrote:
>>> In case I'm not clear, what I'm saying is that I think we can consider
>>> the writable CTE patch ready for commit even though we still have to
>>> decide what its impacts on documentation should be.
>>
>> Whether a patch is ready to commit will be up to the committer
>
> "Ready for Committer" is what I though but failed to type.

*shrug*  Same issue, to some degree.  For a patch of this size, the
difference between "Needs Review" and "Ready for Committer" is maybe
somewhat less than normal.  My point is just that I think there is
work that can be usefully done on this patch by people other than Tom,
even though I believe that ultimately Tom will have to make the call
on whether it goes in.  I don't think that should cause Tom to put off
looking at it himself, but neither do I think that anyone else should
feel like we've accomplished something by labelling it Ready for
Committer.  I'm disappointed that we marked this RfC so early in the
cycle without catching the docs issue; Marko could have started
working on that much sooner if we'd given him that feedback.  Let's
not take our eye off the ball again.

...Robert

On Sun, Feb 7, 2010 at 3:37 PM, Josh Berkus <josh@agliodbs.com> wrote:
>> As between the two, I get the feeling that there is more interest in
>> writeable CTEs.  But that impression might be wrong, since it's an
>> unscientific recollection of discussions on -hackers; which are
>> themselves not representative of anything.
>
> Writeable CTE is definitely the bigger feature.  Effectively, it allows
> people to do in a single query data-transformation operations which
> would have taken a stored procedure before.  Think of it as comparable
> to the introduction of callbacks in Perl for coolness.

Now if I knew what callbacks in Perl were, I'd probably be impressed.
You mean closures?

>> I have not looked at the window functions patch at all, and I haven't
>> looked at the latest version of writeable CTEs, either.  I will try to
>> spend some time on it in the next couple of days.  My feeling about
>> the last version is that it lacked a lot in the documentation
>> department, and also in the comments department.  Since I don't know
>> that code very well, that made it hard for me to assess technical
>> correctness.
>
> Hmmm, that's potentially lethal.  David Fetter has been doing a lot of
> presentations on the feature; surely he could turn them into some
> documentation?  David?

I would be 100% in favor of some more help on the documentation.  I do
plan to reread this patch, but I don't know that I can cover the
amount of work that needs to be done myself, and as you say, lack of
adequate documentation could very well kill this patch.  In fact, I'll
go so far as to say it's one of the most likely reasons why this patch
might not get in.  So any resources we can bring to bear on that issue
would be well spent.

...Robert

2010/2/7 Oleg Bartunov <oleg@sai.msu.su>:
> I understand your complaints. I think, the real problem is that some of us
> live in the part of word with long holidays in December, while we in Russia
> have very long holidays in January. So, about a month we couldn't
> synchronize developers and reviewers.  I'm not sure if we took this into
> account.

Yeah, that definitely made things harder.  I had the feeling when I
started looking at this stuff over Christmas that it was going to take
a really determined and non-stop effort to get it all done, and we
haven't quite had that, either on the reviewing end or on your end.
Your holidays slowed things down, but we also had a quite small pool
of round-robin reviewers for this CF, and I couldn't get anyone to
sign on for knngist.  Mark Cave-Ayland eventually volunteered but that
was relatively late, and then he hasn't posted anything yet because he
got involved in helping with rbtree (which by the way isn't quite
done; we should really try to finish that up).  So I think it was a
combination of things.

By the way, I wish I had your holiday schedule!  Can you send me a few of those?

> In regard to the knngist patch I want to claim, that I and Teodor are here
> and willing to answer any questions.

I really hope that Mark (or someone else) will post a review before
this CommitFest is over.  I believe it is out of reach to get this
committed for this CF, but it would sure be nice to see it get at
least some review.  I would like to review it myself at some point,
but I think right now I need to focus on things that are a little
further along and have a better chance of getting in.

...Robert

2010/2/7 Josh Berkus <josh@agliodbs.com>:
>> As between the two, I get the feeling that there is more interest in
>> writeable CTEs.  But that impression might be wrong, since it's an
>> unscientific recollection of discussions on -hackers; which are
>> themselves not representative of anything.
>
> Writeable CTE is definitely the bigger feature.  Effectively, it allows
> people to do in a single query data-transformation operations which
> would have taken a stored procedure before.  Think of it as comparable
> to the introduction of callbacks in Perl for coolness.

Yes, it's bigger. It's certainly a bigger marketing checkbox item.
That doesn't necessarily make it more useful.

As a comparison point, I've come across a number of cases with clients
where being able to do RANGE BETWEEN on windowing queries would've
been extremely helpful, and where there's no reasonable way to do that
at all today other than to dump all the data off into the application.
Neither of which are exactly pretty or fast.

The similar case for Writable CTEs, I've always been able to wrap it
in a function. Which is nowhere near as nice as having writable CTEs
of course, but the workaround for not having it is less severe.

I certainly wish we could have both, of course... :S

-- Magnus HaganderMe: http://www.hagander.net/Work: http://www.redpill-linpro.com/

Dimitri Fontaine wrote:
> Marko Tiikkaja <marko.tiikkaja@cs.helsinki.fi> writes:
> > The documentation has definitely improved from the last time Robert
> > looked at it, but I fear it still needs some more work.  I'm willing to
> > do that work, but I need something concrete.
> 
> It seems to me documentation is required to get into the source tree
> before beta, and as we see with some other patches it's definitely the
> case even with our newer procedures that some code gets in without its
> documentation properly finished. I guess this amounts to the commiter
> willing to fill up the docs later on.

Eh?  Previously we allowed code to go in with documentation to be
written after feature freeze.  Is this no longer acceptable?

-- 
Alvaro Herrera                                http://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.

On Mon, Feb 8, 2010 at 10:25 AM, Alvaro Herrera
<alvherre@commandprompt.com> wrote:
> Dimitri Fontaine wrote:
>> Marko Tiikkaja <marko.tiikkaja@cs.helsinki.fi> writes:
>> > The documentation has definitely improved from the last time Robert
>> > looked at it, but I fear it still needs some more work.  I'm willing to
>> > do that work, but I need something concrete.
>>
>> It seems to me documentation is required to get into the source tree
>> before beta, and as we see with some other patches it's definitely the
>> case even with our newer procedures that some code gets in without its
>> documentation properly finished. I guess this amounts to the commiter
>> willing to fill up the docs later on.
>
> Eh?  Previously we allowed code to go in with documentation to be
> written after feature freeze.  Is this no longer acceptable?

I don't think we usually allow that for minor features.  For big
things, it's probably more reasonable, but I would think that at least
some effort should be put in before commit.  I'm new here, though, so
I might be all wet.  But I wouldn't want to commit ten patches without
documentation and then have someone come back and say, OK, you
committed 'em, you write the docs.  Or else no one comes back, and I
forget, and it never gets done.

...Robert

Robert Haas <robertmhaas@gmail.com> writes:
> On Mon, Feb 8, 2010 at 10:25 AM, Alvaro Herrera
> <alvherre@commandprompt.com> wrote:
>> Eh?  Previously we allowed code to go in with documentation to be
>> written after feature freeze.  Is this no longer acceptable?
>
> I don't think we usually allow that for minor features.  For big
> things, it's probably more reasonable, but I would think that at least
> some effort should be put in before commit.  I'm new here, though, so
> I might be all wet.  But I wouldn't want to commit ten patches without
> documentation and then have someone come back and say, OK, you
> committed 'em, you write the docs.  Or else no one comes back, and I
> forget, and it never gets done.

Well, traditionnaly, we had Bruce to sort those things out. But in this
case the problem is not so much about writing documentation than
deciding where to put it and what to explain exactly. I think.

Anyway saying the patch can not be considered by a commiter for only
lack of complete documentation is not a policy here, IME. It can happen,
but I would consider it bad news if it were to become a way to force the
release timeframe. What is hard is doing *good* compromises.

Regards,
--
dim

On Mon, Feb 8, 2010 at 11:47 AM, Dimitri Fontaine
<dfontaine@hi-media.com> wrote:
> Robert Haas <robertmhaas@gmail.com> writes:
>> On Mon, Feb 8, 2010 at 10:25 AM, Alvaro Herrera
>> <alvherre@commandprompt.com> wrote:
>>> Eh?  Previously we allowed code to go in with documentation to be
>>> written after feature freeze.  Is this no longer acceptable?
>>
>> I don't think we usually allow that for minor features.  For big
>> things, it's probably more reasonable, but I would think that at least
>> some effort should be put in before commit.  I'm new here, though, so
>> I might be all wet.  But I wouldn't want to commit ten patches without
>> documentation and then have someone come back and say, OK, you
>> committed 'em, you write the docs.  Or else no one comes back, and I
>> forget, and it never gets done.
>
> Well, traditionnaly, we had Bruce to sort those things out. But in this
> case the problem is not so much about writing documentation than
> deciding where to put it and what to explain exactly. I think.
>
> Anyway saying the patch can not be considered by a commiter for only
> lack of complete documentation is not a policy here, IME. It can happen,
> but I would consider it bad news if it were to become a way to force the
> release timeframe. What is hard is doing *good* compromises.

Of course any committer can consider any patch whenever they like,
regardless of how it is marked on commitfest.postgresql.org, right?
And there has been no shortage of committers doing just that; 80%+ of
the reviews for this CommitFest were done by committers.  But I'm not
going to spend the time to write the docs for somebody else's patch
unless I really care about seeing it go in; other committers are free
to do as they like, of course.

...Robert

On 2/8/10 7:31 AM, Robert Haas wrote:
>> Eh?  Previously we allowed code to go in with documentation to be
>> written after feature freeze.  Is this no longer acceptable?

My $0.0201115:

Depends on the feature, I'd say.  If it's sufficiently obvious to test
the feature without full documentation, then sure.  If, however,
reviewers can't adequately test the patch because they don't know all of
the syntax being implemented, then docs are a requirement.

--Josh Berkus

On Feb 8, 2010, at 9:34 AM, Josh Berkus wrote:

>>> Eh?  Previously we allowed code to go in with documentation to be
>>> written after feature freeze.  Is this no longer acceptable?
>
> My $0.0201115:

I think you need to use a NUMERIC type there, as some calculation has lost precision in the float.

Best,

David

On Sun, Feb 7, 2010 at 11:23 PM, Robert Haas <robertmhaas@gmail.com> wrote:
> On Sun, Feb 7, 2010 at 3:37 PM, Josh Berkus <josh@agliodbs.com> wrote:
>>> As between the two, I get the feeling that there is more interest in
>>> writeable CTEs.  But that impression might be wrong, since it's an
>>> unscientific recollection of discussions on -hackers; which are
>>> themselves not representative of anything.
>>
>> Writeable CTE is definitely the bigger feature.  Effectively, it allows
>> people to do in a single query data-transformation operations which
>> would have taken a stored procedure before.  Think of it as comparable
>> to the introduction of callbacks in Perl for coolness.
>
> Now if I knew what callbacks in Perl were, I'd probably be impressed.
> You mean closures?
>
>>> I have not looked at the window functions patch at all, and I haven't
>>> looked at the latest version of writeable CTEs, either.  I will try to
>>> spend some time on it in the next couple of days.  My feeling about
>>> the last version is that it lacked a lot in the documentation
>>> department, and also in the comments department.  Since I don't know
>>> that code very well, that made it hard for me to assess technical
>>> correctness.
>>
>> Hmmm, that's potentially lethal.  David Fetter has been doing a lot of
>> presentations on the feature; surely he could turn them into some
>> documentation?  David?
>
> I would be 100% in favor of some more help on the documentation.  I do
> plan to reread this patch, but I don't know that I can cover the
> amount of work that needs to be done myself, and as you say, lack of
> adequate documentation could very well kill this patch.  In fact, I'll
> go so far as to say it's one of the most likely reasons why this patch
> might not get in.  So any resources we can bring to bear on that issue
> would be well spent.

I'm on board to work on the documentation.  I think with a few hours
of work it should be in a reasonable state.

merlin

On Mon, 8 Feb 2010, Josh Berkus wrote:

> On 2/8/10 7:31 AM, Robert Haas wrote:
>>> Eh?  Previously we allowed code to go in with documentation to be
>>> written after feature freeze.  Is this no longer acceptable?
>
> My $0.0201115:
>
> Depends on the feature, I'd say.  If it's sufficiently obvious to test
> the feature without full documentation, then sure.  If, however,
> reviewers can't adequately test the patch because they don't know all of
> the syntax being implemented, then docs are a requirement.

+1
    Regards,        Oleg
_____________________________________________________________
Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru),
Sternberg Astronomical Institute, Moscow University, Russia
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(495)939-16-83, +007(495)939-23-83

List of holidays by country
http://en.wikipedia.org/wiki/List_of_holidays_by_country

I'm not sure how it's valid, though. In Russia, for example,
russian goverment decreed holidays 1-10 January, 2010. I think next time we
should consider december-january as a half.

Oleg
On Sun, 7 Feb 2010, Robert Haas wrote:

> 2010/2/7 Oleg Bartunov <oleg@sai.msu.su>:
>> I understand your complaints. I think, the real problem is that some of us
>> live in the part of word with long holidays in December, while we in Russia
>> have very long holidays in January. So, about a month we couldn't
>> synchronize developers and reviewers.  I'm not sure if we took this into
>> account.
>
> Yeah, that definitely made things harder.  I had the feeling when I
> started looking at this stuff over Christmas that it was going to take
> a really determined and non-stop effort to get it all done, and we
> haven't quite had that, either on the reviewing end or on your end.
> Your holidays slowed things down, but we also had a quite small pool
> of round-robin reviewers for this CF, and I couldn't get anyone to
> sign on for knngist.  Mark Cave-Ayland eventually volunteered but that
> was relatively late, and then he hasn't posted anything yet because he
> got involved in helping with rbtree (which by the way isn't quite
> done; we should really try to finish that up).  So I think it was a
> combination of things.
>
> By the way, I wish I had your holiday schedule!  Can you send me a few of those?
>
>> In regard to the knngist patch I want to claim, that I and Teodor are here
>> and willing to answer any questions.
>
> I really hope that Mark (or someone else) will post a review before
> this CommitFest is over.  I believe it is out of reach to get this
> committed for this CF, but it would sure be nice to see it get at
> least some review.  I would like to review it myself at some point,
> but I think right now I need to focus on things that are a little
> further along and have a better chance of getting in.
>
> ...Robert
>
    Regards,        Oleg
_____________________________________________________________
Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru),
Sternberg Astronomical Institute, Moscow University, Russia
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(495)939-16-83, +007(495)939-23-83

2010/2/8 Oleg Bartunov <oleg@sai.msu.su>:
> List of holidays by country
> http://en.wikipedia.org/wiki/List_of_holidays_by_country
>
> I'm not sure how it's valid, though. In Russia, for example,
> russian goverment decreed holidays 1-10 January, 2010. I think next time we
> should consider december-january as a half.

Oh, I wasn't asking for a list of your holidays - I was just wishing
that I had as many as it sounds like you do.  :-)

...Robert