Thread: Releasing in September

Many people where happy with our consistent releasing major releases in
September, e.g. 9.0 to 9.3:
9.5   2016-01-079.4   2014-12-189.3   2013-09-09 <--9.2   2012-09-10 <--9.1   2011-09-12 <--9.0   2010-09-20 <--8.4
2009-07-018.3  2008-02-048.2   2006-12-058.1   2005-11-088.0   2005-01-19
 

We have gotten off of that cycle in the last two major releases, and
this isn't going to improve as long as we have commitfests starting
after January.  In the September-release years, we used to start
preparing for beta in mid-February or early March.  However, for 9.5 we
had our last commitfest _start_ in mid-February, and it didn't end until
mid-May.  For 9.6 we have a commitfest starting in March 1.  This will
never allow a September release.

Our current 9.5/9.6 timing looks more like the 8.X series of release
dates.  Everyone might be fine with that, but we had better be prepared
for November-February major release dates going forward.

--  Bruce Momjian  <bruce@momjian.us>        http://momjian.us EnterpriseDB
http://enterprisedb.com

+ As you are, so once was I. As I am, so you will be. +
+ Roman grave inscription                             +

On 2016-01-20 10:40:14 -0500, Bruce Momjian wrote:
> We have gotten off of that cycle in the last two major releases, and
> this isn't going to improve as long as we have commitfests starting
> after January.

I think this has very little to do with commitfest schedules, and much
more with the "early" forking of the new version branch. For both 9.4
and 9.5 we essentially spent a couple months twiddling our thumbs.

On Wed, Jan 20, 2016 at 04:48:16PM +0100, Andres Freund wrote:
> On 2016-01-20 10:40:14 -0500, Bruce Momjian wrote:
> > We have gotten off of that cycle in the last two major releases, and
> > this isn't going to improve as long as we have commitfests starting
> > after January.
> 
> I think this has very little to do with commitfest schedules, and much
> more with the "early" forking of the new version branch. For both 9.4
> and 9.5 we essentially spent a couple months twiddling our thumbs.

Well, when the commitfest extends to May, you are not going to fork the
next release, and since we haven't forked by PGCon in Ottawa, we try to
get in five commitfest, instead of four, and we then get the schedule we
have now.

--  Bruce Momjian  <bruce@momjian.us>        http://momjian.us EnterpriseDB
http://enterprisedb.com

+ As you are, so once was I. As I am, so you will be. +
+ Roman grave inscription                             +

On 01/20/2016 07:40 AM, Bruce Momjian wrote:

> Our current 9.5/9.6 timing looks more like the 8.X series of release
> dates.  Everyone might be fine with that, but we had better be prepared
> for November-February major release dates going forward.

Bruce,

Thank you for bringing this up it is a good time to discuss it. I think 
we have a number of things we need to consider when it comes to releases:

1. When we release
2. What we release
3. When not to release

#1 September is actually a great time to release. However, I think that 
we are going to run into trouble keeping that time frame. The reality is 
PostgreSQL of today is a lot more complex than the PostgreSQL of even 3 
years ago.

That means that unless we are willing to have stunted releases, we are 
going to miss release dates. We need to be able to say, "Ompf, no whiz 
bang features this release, mostly polish" or "These whiz bang features 
need more testing, we are pushing our release". If we aren't willing to 
do that then we need to reconsider having a fixed release date and 
perhaps start a release time frame (9.6 will be released Q4 2016).

#2 This is a longer topic. I have been stewing in my head about releases 
for years. I have even brought up the idea of an Ubuntu style release 
cycle on list once or twice. The more I think about it, the more I think 
this can help our community. We essentially would have two types of 
releases:

STS:

* Supported for 1 release cycle plus 6 months (18-24 months)
* Inline upgrades supported

LTS:

* Supported for standard 5 years
* Is allowed to break binary format from STS but not previous LTS. This 
allows two LTS versions per 5 year support cycle

There is more to say on this but this is already a long reply.

#3 The fact that 9.5 was delayed is a good thing, not a bad one. I 
believe we need to be much more willing to push a button that says, "It 
isn't done." Which goes back to the idea in #1 of having a release 
"window" versus date. We aren't a company. We don't need to adhere to an 
exact release schedule.

Sincerely,

Joshua D. Drake






-- 
Command Prompt, Inc.                  http://the.postgres.company/                     +1-503-667-4564
PostgreSQL Centered full stack support, consulting and development.

On Wed, Jan 20, 2016 at 10:48 AM, Andres Freund <andres@anarazel.de> wrote:
> On 2016-01-20 10:40:14 -0500, Bruce Momjian wrote:
>> We have gotten off of that cycle in the last two major releases, and
>> this isn't going to improve as long as we have commitfests starting
>> after January.
>
> I think this has very little to do with commitfest schedules, and much
> more with the "early" forking of the new version branch. For both 9.4
> and 9.5 we essentially spent a couple months twiddling our thumbs.

It's certainly true that we twiddled our thumbs quite a bit about
getting 9.5 ready to ship.  However, the old process where nobody
could get anything committed for six months out of the year blew
chunks, too.  Personally, I think that the solution is to cut off the
last CommitFest a lot sooner, and then reopen the tree for the next
release as soon as possible.  But this never works, because there are
always patches we want to slip in late.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Wed, Jan 20, 2016 at 10:55:07AM -0500, Robert Haas wrote:
> On Wed, Jan 20, 2016 at 10:48 AM, Andres Freund <andres@anarazel.de> wrote:
> > I think this has very little to do with commitfest schedules, and much
> > more with the "early" forking of the new version branch. For both 9.4
> > and 9.5 we essentially spent a couple months twiddling our thumbs.
> 
> It's certainly true that we twiddled our thumbs quite a bit about
> getting 9.5 ready to ship.  However, the old process where nobody
> could get anything committed for six months out of the year blew
> chunks, too.  Personally, I think that the solution is to cut off the
> last CommitFest a lot sooner, and then reopen the tree for the next
> release as soon as possible.  But this never works, because there are
> always patches we want to slip in late.

The bottom line is we can't sustain five commitfests and stay on time
--- we need to go back to four, which I think is what we used to do.  We
twiddled our thumbs back in the September-release years too, but had
consistency because twiddling was built into the schedule.

--  Bruce Momjian  <bruce@momjian.us>        http://momjian.us EnterpriseDB
http://enterprisedb.com

+ As you are, so once was I. As I am, so you will be. +
+ Roman grave inscription                             +

On 2016-01-20 10:55:07 -0500, Robert Haas wrote:
> It's certainly true that we twiddled our thumbs quite a bit about
> getting 9.5 ready to ship.  However, the old process where nobody
> could get anything committed for six months out of the year blew
> chunks, too.  Personally, I think that the solution is to cut off the
> last CommitFest a lot sooner, and then reopen the tree for the next
> release as soon as possible.  But this never works, because there are
> always patches we want to slip in late.

Said twiddling seems to largely happened from July to December. In which
the other branch was open, and no 9.5 commitfest was happening. If we
move the commitfests to earlier, but still have a half year of nothing
happening, we're still in a bad situation.

FWIW, looking at the last few commitfests, aside heroic and
unsustainable efforts by individual CF managers, I haven't noticed any
effect of when fests started/stopped. Aside from a short time increase
in unfinished patches being posted the day before the next CFs starts.

Andres

<p dir="ltr"><br /> On Jan 20, 2016 5:03 PM, "Andres Freund" <<a
href="mailto:andres@anarazel.de">andres@anarazel.de</a>>wrote:<br /> ><br /> > On 2016-01-20 10:55:07 -0500,
RobertHaas wrote:<br /> > > It's certainly true that we twiddled our thumbs quite a bit about<br /> > >
getting9.5 ready to ship.  However, the old process where nobody<br /> > > could get anything committed for six
monthsout of the year blew<br /> > > chunks, too.  Personally, I think that the solution is to cut off the<br />
>> last CommitFest a lot sooner, and then reopen the tree for the next<br /> > > release as soon as
possible. But this never works, because there are<br /> > > always patches we want to slip in late.<br /> ><br
/>> Said twiddling seems to largely happened from July to December. In which<br /> > the other branch was open,
andno 9.5 commitfest was happening. If we<br /> > move the commitfests to earlier, but still have a half year of
nothing<br/> > happening, we're still in a bad situation.<br /> ><br /> > FWIW, looking at the last few
commitfests,aside heroic and<br /> > unsustainable efforts by individual CF managers, I haven't noticed any<br />
>effect of when fests started/stopped. Aside from a short time increase<br /> > in unfinished patches being
postedthe day before the next CFs starts.<br /><p dir="ltr">Yeah, we seem to be firmly stuck at two month long
commitfestsstarted every two months. The plan was for them to be one month.. <p dir="ltr">Maybe we should try just very
drasticallycutting them at one month and bumping everything left. No questions asked, no extra time for anybody.
Regardlessof if it's the first or the last commitfest. <p dir="ltr">Just to see what happens. Because what we are doing
nowclearly doesn't work.. <p dir="ltr">/Magnus  

Magnus Hagander <magnus@hagander.net> writes:
> On Jan 20, 2016 5:03 PM, "Andres Freund" <andres@anarazel.de> wrote:
>> FWIW, looking at the last few commitfests, aside heroic and
>> unsustainable efforts by individual CF managers, I haven't noticed any
>> effect of when fests started/stopped. Aside from a short time increase
>> in unfinished patches being posted the day before the next CFs starts.

> Yeah, we seem to be firmly stuck at two month long commitfests started
> every two months. The plan was for them to be one month..

> Maybe we should try just very drastically cutting them at one month and
> bumping everything left. No questions asked, no extra time for anybody.
> Regardless of if it's the first or the last commitfest.

> Just to see what happens. Because what we are doing now clearly doesn't
> work..

I do not think commitfest length is the problem (though surely it's not
working as intended).  What happened with 9.5 is we forked the 9.6
development branch on June 30th, with the expectation of releasing in
September, and then couldn't release in September because nobody had done
any significant amount of stabilization work.  Instead we had the 2015-07
commitfest.  And the 2015-09 commitfest.  And the 2015-11 commitfest.

We will not get back to on-schedule releases unless we can keep -hackers
working on release testing/stabilization when it's time to do that,
rather than being distracted by shiny new stuff going into the next
release.

I'd suggest that the easiest process fix is to not fork a new devel branch
until we reach, say, RC status.

And that does mean losing at least one commitfest per release cycle ---
but it would be the currently-first one, not the currently-last one.
        regards, tom lane

On 2016-01-20 11:19:28 -0500, Tom Lane wrote:
> We will not get back to on-schedule releases unless we can keep -hackers
> working on release testing/stabilization when it's time to do that,
> rather than being distracted by shiny new stuff going into the next
> release.

Agreed. I'll note that the linux kernel used to have a similar
problem. Now there's a two week integration and a 10 week stabilization
period, with occasionally a week added/subtracted. If a feature patch is
submitted for integration (not just parallel review) into the main
branch outside the merge window you get yelled at.  Now, we're at least
two magnitudes smaller, so not everything there applies to us. But I
think looking over that fence, to see how they scaled from smaller where
we are now, to way way bigger, isn't a bad idea.


I think one thing we should work on, is being absolutely religious about
requiring, say, 2 reviews for every nontrivial contribution.  We
currently seem to have a significantly increased submission rate, and at
the same time the number of reviews per patch has gone down
substantially.  I think the "honor" system has failed in that regard.


Andres

On Wed, Jan 20, 2016 at 07:53:25AM -0800, Joshua Drake wrote:
> #2 This is a longer topic. I have been stewing in my head about
> releases for years. I have even brought up the idea of an Ubuntu
> style release cycle on list once or twice. The more I think about
> it, the more I think this can help our community. We essentially
> would have two types of releases:
> 
> STS:
> 
> * Supported for 1 release cycle plus 6 months (18-24 months)
> * Inline upgrades supported
> 
> LTS:
> 
> * Supported for standard 5 years
> * Is allowed to break binary format from STS but not previous LTS.
> This allows two LTS versions per 5 year support cycle

I just don't buy the Ubuntu release model for our database.  Ubuntu is
trying to balance hot features vs stability, while we are really focused
on stability, similar to Debian.

--  Bruce Momjian  <bruce@momjian.us>        http://momjian.us EnterpriseDB
http://enterprisedb.com

+ As you are, so once was I. As I am, so you will be. +
+ Roman grave inscription                             +

On Wed, Jan 20, 2016 at 11:19 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> I do not think commitfest length is the problem (though surely it's not
> working as intended).  What happened with 9.5 is we forked the 9.6
> development branch on June 30th, with the expectation of releasing in
> September, and then couldn't release in September because nobody had done
> any significant amount of stabilization work.  Instead we had the 2015-07
> commitfest.  And the 2015-09 commitfest.  And the 2015-11 commitfest.

But I'm not very sure that we're talking about the same set of people
here.  If we're going to go to a system where nobody's allowed to
commit anything for the next release until the current release is
finalized, then we'd better have some procedure for making sure that
happens relatively quickly.  And the procedure can't be that the
people who are hot to get started on the next release have to bat
cleanup for the people who don't have time to fix the bugs they
introduced previously.  Because *that* would be manifestly unfair.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Wed, Jan 20, 2016 at 11:53:32AM -0500, Robert Haas wrote:
> On Wed, Jan 20, 2016 at 11:19 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> > I do not think commitfest length is the problem (though surely it's not
> > working as intended).  What happened with 9.5 is we forked the 9.6
> > development branch on June 30th, with the expectation of releasing in
> > September, and then couldn't release in September because nobody had done
> > any significant amount of stabilization work.  Instead we had the 2015-07
> > commitfest.  And the 2015-09 commitfest.  And the 2015-11 commitfest.
> 
> But I'm not very sure that we're talking about the same set of people
> here.  If we're going to go to a system where nobody's allowed to
> commit anything for the next release until the current release is
> finalized, then we'd better have some procedure for making sure that
> happens relatively quickly.  And the procedure can't be that the
> people who are hot to get started on the next release have to bat
> cleanup for the people who don't have time to fix the bugs they
> introduced previously.  Because *that* would be manifestly unfair.

Unfair or not, that is probably the effect.  I can also see people
publishing git trees to avoid this roadblock.

--  Bruce Momjian  <bruce@momjian.us>        http://momjian.us EnterpriseDB
http://enterprisedb.com

+ As you are, so once was I. As I am, so you will be. +
+ Roman grave inscription                             +

On 2016-01-20 11:53:32 -0500, Robert Haas wrote:
> But I'm not very sure that we're talking about the same set of people
> here.  If we're going to go to a system where nobody's allowed to
> commit anything for the next release until the current release is
> finalized, then we'd better have some procedure for making sure that
> happens relatively quickly.  And the procedure can't be that the
> people who are hot to get started on the next release have to bat
> cleanup for the people who don't have time to fix the bugs they
> introduced previously.  Because *that* would be manifestly unfair.

Yes, that's a problem. But I don't see how we can avoid dealing with it
directly. Avoiding being stalled by unfixed bugs of others by having a
separate branch to continue working, inevitably leads to delayed
releases.  If people don't fix the issues in time, there needs to be
direct pushback, leading to much less stuff getting in next time round.

Robert Haas <robertmhaas@gmail.com> writes:
> But I'm not very sure that we're talking about the same set of people
> here.  If we're going to go to a system where nobody's allowed to
> commit anything for the next release until the current release is
> finalized, then we'd better have some procedure for making sure that
> happens relatively quickly.  And the procedure can't be that the
> people who are hot to get started on the next release have to bat
> cleanup for the people who don't have time to fix the bugs they
> introduced previously.  Because *that* would be manifestly unfair.

You're assuming that the people who are hot to get started on the next
release are different from the people who don't have time to fix the bugs
they introduced previously.  IME it's mostly the same people.
        regards, tom lane

On 01/20/2016 08:51 AM, Bruce Momjian wrote:
> On Wed, Jan 20, 2016 at 07:53:25AM -0800, Joshua Drake wrote:
>> #2 This is a longer topic. I have been stewing in my head about
>> releases for years. I have even brought up the idea of an Ubuntu
>> style release cycle on list once or twice. The more I think about
>> it, the more I think this can help our community. We essentially
>> would have two types of releases:
>>
>> STS:
>>
>> * Supported for 1 release cycle plus 6 months (18-24 months)
>> * Inline upgrades supported
>>
>> LTS:
>>
>> * Supported for standard 5 years
>> * Is allowed to break binary format from STS but not previous LTS.
>> This allows two LTS versions per 5 year support cycle
>
> I just don't buy the Ubuntu release model for our database.  Ubuntu is
> trying to balance hot features vs stability, while we are really focused
> on stability, similar to Debian.

I understand but I think we are missing out on an opportunity here. 
Notice that the shorter release cycle for STS will actually make some 
things easier. Including:
 * Increased test base (just like Fedora/Ubuntu) * Increased early adopter testing (that is what early adopting is 
really about for us anyway) * Decreased concerns about upgrades and ability to extend upgrade status.

I am not in any way suggesting that this is a slam dunk but I do think 
something along these lines can really bring forth a lot of 
possibilities. Here is another example:

pg_audit was pulled from core, did it have to be? I can't say. I didn't 
read the code. I was a proactive member stating we could pull it though 
because it was an extension.

However, if it was an STS release it was going into we could be a little 
more relaxed and say, "O.k. let's get this into the wild for some 
general user testing".

There are a lot of people (think about the multixact problem) that will 
run any software because it is new. Let's put the proper disclaimers on 
there and let them run it.

Sincerely,

Joshua D. Drake






-- 
Command Prompt, Inc.                  http://the.postgres.company/                     +1-503-667-4564
PostgreSQL Centered full stack support, consulting and development.

On 01/20/2016 09:03 AM, Andres Freund wrote:
> If people don't fix the issues in time, there needs to be
> direct pushback, leading to much less stuff getting in next time round.
>

We have been slowly moving to a more dictator based release anyway. It 
used to be that we released "when it's done", then we moved to 
commitfest, and now we are having yet another discussion on the same 
topic of how to manage all of this.

It seems clear that we need to be able to say:

* "Your feature is awesome. Sorry, you are out of time"

Life is hard sometimes, we don't always get what we want.

Sincerely,

JD



-- 
Command Prompt, Inc.                  http://the.postgres.company/                     +1-503-667-4564
PostgreSQL Centered full stack support, consulting and development.

On 2016-01-20 09:15:01 -0800, Joshua D. Drake wrote:
> On 01/20/2016 09:03 AM, Andres Freund wrote:
> >If people don't fix the issues in time, there needs to be
> >direct pushback, leading to much less stuff getting in next time round.
> >
> 
> We have been slowly moving to a more dictator based release anyway. It used
> to be that we released "when it's done", then we moved to commitfest, and
> now we are having yet another discussion on the same topic of how to manage
> all of this.
> 
> It seems clear that we need to be able to say:
> 
> * "Your feature is awesome. Sorry, you are out of time"

That's not really related to the discussion here tho? The debated
problem is things that have already been committed that have have bugs,
preventing beta/rc releases from being made. In some cases in the last
releases such bugs were in features integrated very early in the cycle.

Magnus Hagander <magnus@hagander.net> writes:
> On Wed, Jan 20, 2016 at 5:19 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> I do not think commitfest length is the problem (though surely it's not
>> working as intended).  What happened with 9.5 is we forked the 9.6

> I agree that it's not the same problem. I do believe that it is *a* problem
> though, and a fairly significant one too. Because there's *never* any
> downtime from CF mode, regardless of where in the cycle we are.

True, we've been failing badly on the intention that there would be time
off from CF mode, and I'd like to see a fix for that.  I do not think it's
directly related to the can't-get-a-release-out problem.

I'm not really sure why we've allowed CFs to drift on, though.  Can't we
just arbitrarily decree them closed on the last day of the month?  And
push unfinished work to the next one?  Admittedly, this probably doesn't
work for the last CF of a release cycle, but that one's always been a
special case.
        regards, tom lane

On 01/20/2016 09:17 AM, Andres Freund wrote:
> On 2016-01-20 09:15:01 -0800, Joshua D. Drake wrote:
>> On 01/20/2016 09:03 AM, Andres Freund wrote:
>>> If people don't fix the issues in time, there needs to be
>>> direct pushback, leading to much less stuff getting in next time round.
>>>
>>
>> We have been slowly moving to a more dictator based release anyway. It used
>> to be that we released "when it's done", then we moved to commitfest, and
>> now we are having yet another discussion on the same topic of how to manage
>> all of this.
>>
>> It seems clear that we need to be able to say:
>>
>> * "Your feature is awesome. Sorry, you are out of time"
>
> That's not really related to the discussion here tho? The debated
> problem is things that have already been committed that have have bugs,
> preventing beta/rc releases from being made. In some cases in the last
> releases such bugs were in features integrated very early in the cycle.

Sorry if I was causing noise. I was under the impression that the 
discussion was more than just things that already have bugs, but also 
how the commitfests were working (scheduling etc..). I admit, I may have 
grabbed your comment out of an unrelated portion of the thread.

JD


-- 
Command Prompt, Inc.                  http://the.postgres.company/                     +1-503-667-4564
PostgreSQL Centered full stack support, consulting and development.

On 2016-01-20 12:18:28 -0500, Tom Lane wrote:
> I'm not really sure why we've allowed CFs to drift on, though.  Can't we
> just arbitrarily decree them closed on the last day of the month?  And
> push unfinished work to the next one?  Admittedly, this probably doesn't
> work for the last CF of a release cycle, but that one's always been a
> special case.

I think the problem there is the assumption that each CF entry deserves
some amount of feedback. It's not a big problem to close entries of
patches that have gotten a fair amount, but it's pretty common to have a
set of entries that have barely gotten any review. Usually either
because they're perceived as too boring (fair enough), because they're
too complex for the majority of non-busy people (uhhh), or because
they're seen as claimed.

Andres

> I admit, I may have grabbed your comment out of an unrelated portion
> of the thread.

Ceterum censeo Carthaginem esse delendam.

On Wed, Jan 20, 2016 at 5:29 PM, Andres Freund <andres@anarazel.de> wrote:
> I think one thing we should work on, is being absolutely religious about
> requiring, say, 2 reviews for every nontrivial contribution.  We
> currently seem to have a significantly increased submission rate, and at
> the same time the number of reviews per patch has gone down
> substantially.  I think the "honor" system has failed in that regard.

Good point, totally agree.

So the project should try to think of new ideas on how to incentivise reviewing?

I have three ideas so far:

1. The way I see it, the honor system is based on being mentioned
in the commit message and the release notes.

Authors are always mentioned in the release notes,
but I see reviewers are mostly only mentioned in the commit messages.

Maybe more skilled developers would think it's cool to do reviewing
if they were "paid" by also being mentioned in the release notes?

At least some skilled people would probably be motivated by it,
in addition to the good feeling of doing something just because it's
fun or important.

2. Currently "Needs review" can mean a patch is in any of the phases
(Submission -> Usability -> Feature test -> Performance review ->
Coding review -> Architecture review -> Review review).
If you as a reviewer don't even know if the patch will be accepted and
committed by a committer,
there is a risk review-time will be wasted until the patch has been
accepted by the community.

I suggest we inject a new intermediate optional step "Needs Code
Review" to mean the feature has been discussed on pghackers
and there is a consensus among committers that they at least agree the
feature is something desired,
and that someone has at least reviewed the patch applies and the
feature works like expected.
And maybe renaming the "Needs review" step to "Needs Usability Review"
(or some other word to capture the Submission -> Usability -> Feature
test -> Performance review, phase before the Code review phase).

Then you as a reviewer would be confident the feature will get
accepted as long as the code is correct,
and the time you spent on code-reviewing will not be wasted.

3. For those who are partly motivated by money,
maybe this idea would work too to some extent:

Since Trustly's Bug Country program [1] is aimed at motivating people
to find bugs in HEAD hasn't produced a single report so far,
maybe we should extend it to also cover reviews of commits
marked as "Ready For Committer" that shed light on a problem
that could have caused data corruption,
preventing a bad commit from being committed.

If we suddenly get hundreds of new reviewers who find hundreds of bugs
in uncommitted code, then maybe we have to change the terms again,
but I doubt neither of those will happen.

If people think this is a good idea, I can discuss internally if we can change
the conditions of the bug country program accordingly.

[1] https://trustly.com/se/postgresql-bug-bounty/

Magnus Hagander <magnus@hagander.net> writes:
> That's pretty much what I suggested :)

> Except that we need to do it for the last one as well. With the only
> exception that the last one might be a bit longer. But the fact is that the
> recent of CFs *and* releases, we've taken the approach of closing the CF
> when everything in it is done or explicitly reviewed-and-bumped, and tried
> really really hard not to bump things because nobody had time to look at
> them. That's what I'm suggesting we change, and actually just cut them.
> Yes, one of the reasons for the CFs was to allow people a fair chance to
> get reviewed.But given that there isn't actually a deadline in practice
> doesn't help with that.

Yeah.  It's certainly unfair if someone's patch doesn't get reviewed,
but there are only 24 hours in a day, and we have a limited pool of
reviewer and committer manpower.  I think we just have to say that
sometimes life is unfair.

I think it might also be a good idea if we could somehow distinguish
"nobody had time for that yet" from "nobody is interested", with an eye
to flat-out rejecting patches that no one cares enough about to review.
Maybe we could reduce the workload by inventing some kind of up-front
filter whereby a patch isn't even a candidate to get reviewed unless, say,
at least two people besides the author say "this sounds like it's worth
pursuing".

One of the other things I do not like about the current CF process is that
it's created a default assumption that most submitted patches should get
accepted eventually.  It is *very* hard to reject a patch altogether in
the CF process: you more or less have to show cause why it should not get
accepted.  That default is backwards.  Maybe this isn't the process' fault
exactly, but it sure seems like that's how we approach patches now.
        regards, tom lane

On 01/20/2016 09:31 AM, Joel Jacobson wrote:
> On Wed, Jan 20, 2016 at 5:29 PM, Andres Freund <andres@anarazel.de> wrote:
>> I think one thing we should work on, is being absolutely religious about
>> requiring, say, 2 reviews for every nontrivial contribution.  We
>> currently seem to have a significantly increased submission rate, and at
>> the same time the number of reviews per patch has gone down
>> substantially.  I think the "honor" system has failed in that regard.
>
> Good point, totally agree.
>
> So the project should try to think of new ideas on how to incentivise reviewing?
>
> I have three ideas so far:

4. Submit a patch, review a patch.

Don't review patches? Don't submit patches.

JD

-- 
Command Prompt, Inc.                  http://the.postgres.company/                     +1-503-667-4564
PostgreSQL Centered full stack support, consulting and development.

On 2016-01-20 09:42:28 -0800, Joshua D. Drake wrote:
> Don't review patches? Don't submit patches.

Absolutely. While I think encouraging reviewers is a good thing, it
seems independent of requiring contributors to do quid-pro-quo
reviews. There'll always be people too busy or uninterested in reviewing
riding free on others work otherwise.

Obviously a drive by bugfix is something different than an actual new
feature.

On Jan 20, 2016, at 9:42 AM, Joshua D. Drake <jd@commandprompt.com> wrote:

> 4. Submit a patch, review a patch.
>
> Don't review patches? Don't submit patches.

There will always be patches desirable-enough that they will be reviewed whether or not the submitter reviewed other
patches.

And there will often be patches that generate so little interest that they’ll never be reviewed no matter how many
otherpatches the submitter reviews. 

That said, it’s not a bad heuristic, and I suspect that someone who reviews patches is more likely to get their patch
reviewed.But obviously there are no guarantees. 

Best,

David

Magnus Hagander wrote:

> Except that we need to do it for the last one as well. With the only
> exception that the last one might be a bit longer. But the fact is that the
> recent of CFs *and* releases, we've taken the approach of closing the CF
> when everything in it is done or explicitly reviewed-and-bumped, and tried
> really really hard not to bump things because nobody had time to look at
> them. That's what I'm suggesting we change, and actually just cut them.
> Yes, one of the reasons for the CFs was to allow people a fair chance to
> get reviewed.But given that there isn't actually a deadline in practice
> doesn't help with that.

We have a rule in the commitfest process, which I mentioned in some
other thread yesterday, which we have never enforced as far as I
remember -- which is that the last commitfest only accepts patches that
have already been submitted to previous commitfests.  If we were to
enforce that one, I think we put a limit to the size of the last
commitfest, reducing the amount of work necessary to close it.

I definitely think we should be stricter about closing each commitfest
on the last day of the month.  To improve on the fairness front we could
also be stricter about giving priority to reviewing patches that have
been punted from previous commitfest.

-- 
Álvaro Herrera                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On 2016-01-20 09:48:24 -0800, David E. Wheeler wrote:
> There will always be patches desirable-enough that they will be reviewed whether or not the submitter reviewed other
patches.

I think that's actually getting less and less true. By now the really
desirable-enough features imply so much work by reviewer and committers,
over a prolonged time, that they're imo unlikely to be picked up just
because they're desirable, even when the author doesn't play by the
rules.  I don't think an exceptional case or two is particularly
important.

On 01/20/2016 09:48 AM, David E. Wheeler wrote:
> On Jan 20, 2016, at 9:42 AM, Joshua D. Drake <jd@commandprompt.com> wrote:
>
>> 4. Submit a patch, review a patch.
>>
>> Don't review patches? Don't submit patches.
>
> There will always be patches desirable-enough that they will be reviewed whether or not the submitter reviewed other
patches.
>

Correct.

> And there will often be patches that generate so little interest that they’ll never be reviewed no matter how many
otherpatches the submitter reviews.
 
>

Correct.

> That said, it’s not a bad heuristic, and I suspect that someone who reviews patches is more likely to get their patch
reviewed.But obviously there are no guarantees.
 

Well a good portion of humanity is built on karma. I do for you, you do 
for me. It isn't explicit, that is just bad taste but there is an 
implicit desire that gets built up to help people when you are also 
being helped.

Sincerely,

JD



-- 
Command Prompt, Inc.                  http://the.postgres.company/                     +1-503-667-4564
PostgreSQL Centered full stack support, consulting and development.

Joshua D. Drake wrote:

> 4. Submit a patch, review a patch.
> 
> Don't review patches? Don't submit patches.

Here's one area where the commitfest app could help the CFM.  I would
like to have a report that shows, for each person, the list of patches
where they are author, and the list of patches where they are reviewer.
Then I can check his reviewer score by skimming a couple of threads and
say "you don't have enough author credits, go away".

-- 
Álvaro Herrera                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

<p dir="ltr"><br /> On Jan 20, 2016 18:57, "Alvaro Herrera" <<a
href="mailto:alvherre@2ndquadrant.com">alvherre@2ndquadrant.com</a>>wrote:<br /> ><br /> > Joshua D. Drake
wrote:<br/> ><br /> > > 4. Submit a patch, review a patch.<br /> > ><br /> > > Don't review
patches?Don't submit patches.<br /> ><br /> > Here's one area where the commitfest app could help the CFM.  I
would<br/> > like to have a report that shows, for each person, the list of patches<br /> > where they are
author,and the list of patches where they are reviewer.<br /> > Then I can check his reviewer score by skimming a
coupleof threads and<br /> > say "you don't have enough author credits, go away".<p dir="ltr">I assume you mean a
summaryreview? You can already use the filter feature to get this information if you want to look at an individual
contributor,but I'm guessing you're looking for a global overview? <p dir="ltr">And you want the actual patches, and
notjust a count? <p dir="ltr">/Magnus <br /> 

Magnus Hagander wrote:
> On Jan 20, 2016 18:57, "Alvaro Herrera" <alvherre@2ndquadrant.com> wrote:

> > Here's one area where the commitfest app could help the CFM.  I would
> > like to have a report that shows, for each person, the list of patches
> > where they are author, and the list of patches where they are reviewer.
> > Then I can check his reviewer score by skimming a couple of threads and
> > say "you don't have enough author credits, go away".
> 
> I assume you mean a summary review? You can already use the filter feature
> to get this information if you want to look at an individual contributor,
> but I'm guessing you're looking for a global overview?

Yes, a summary.  I tried using the filters but it's too cumbersome.

> And you want the actual patches, and not just a count?

Yes, the actual patches, so that I can go and see the threads.
Otherwise it's easy to mark oneself as reviewer and not actually review
anything ...

-- 
Álvaro Herrera                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On 2016-01-20 19:00:05 +0100, Magnus Hagander wrote:
> And you want the actual patches, and not just a count?

I think the actual patches makes sense, because reviewing one 200KB
patch obviously is something different than reviewing 3 onelinesrs.

Andres Freund <andres@anarazel.de> writes:
> On 2016-01-20 09:48:24 -0800, David E. Wheeler wrote:
>> There will always be patches desirable-enough that they will be reviewed whether or not the submitter reviewed other
patches.

> I think that's actually getting less and less true. By now the really
> desirable-enough features imply so much work by reviewer and committers,
> over a prolonged time, that they're imo unlikely to be picked up just
> because they're desirable, even when the author doesn't play by the
> rules.  I don't think an exceptional case or two is particularly
> important.

Thinking about that crystallized something for me that maybe is worth
considering.  There is a difference between big patches (say, the
size of the GROUPING SETS one) and little patches (say, most of the
tab-completion fixes we get).  We're currently trying to apply the
same process and criteria to both cases, and I'm thinking that that is
a bad idea.  I am not sure what exactly ought to be different about
them, but probably something should.  I think for small patches, we are
using the CF app mostly to be sure things don't fall through the cracks,
but maybe we don't need the whole process otherwise.
        regards, tom lane

On Wed, Jan 20, 2016 at 12:03 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Robert Haas <robertmhaas@gmail.com> writes:
>> But I'm not very sure that we're talking about the same set of people
>> here.  If we're going to go to a system where nobody's allowed to
>> commit anything for the next release until the current release is
>> finalized, then we'd better have some procedure for making sure that
>> happens relatively quickly.  And the procedure can't be that the
>> people who are hot to get started on the next release have to bat
>> cleanup for the people who don't have time to fix the bugs they
>> introduced previously.  Because *that* would be manifestly unfair.
>
> You're assuming that the people who are hot to get started on the next
> release are different from the people who don't have time to fix the bugs
> they introduced previously.  IME it's mostly the same people.

The last common commit between master and REL9_5_STABLE is dated June
30th.  Here is the open items list as of that date:

https://wiki.postgresql.org/index.php?title=PostgreSQL_9.5_Open_Items&oldid=25373

The only thing on that list that I had anything to do with is "Foreign
join pushdown vs EvalPlanQual".  And that only would have mattered to
people who want to do out-of-core FDW join pushdown against 9.5 and
aren't comfortable restricting the feature to queries without
rowmarks, which is probably nobody, and "PGXS "check" target forcing
an install ?", which was a very minor issue that I didn't create but
did help fix.  Similarly, I see nothing on that list that could be
attributed to, say, you.

I hate to name names, but it seems to me that where things came off
the rails for 9.5 is that (1) the row-level security feature was
absolutely riddled with bugs and that those bugs were not fixed in
anything like timely fashion; (2) the commit timestamp stuff was
pretty badly broken, too; and (3) Andres's multixact reworks landed
quite late and IMHO were not safe enough to back-patch, yet they
needed to be in the release.  In my view, the last major fix for (1)
was in 7d8db3e8f37aec9d252353904e77381a18a2fa9f on September 30th, for
(2) was in commit 69e7235c93e2965cc0e17186bd044e4c54997c19 on December
11th, and for (3) was in aa29c1ccd9f785f9365809f5133e5491acc7ae53 on
September 26th.  I think the row-security stuff is particularly bad
because that feature was originally committed on September 19, *2014*.
It's not good when you commit a security feature and are still
redefining the way that the privilege system interacts with that
feature a year later.

Also note the dates.  We put out beta1 in the beginning of October,
right after those major fixes for (1) and (3) in late September.  And
we put out rc1 just after (2) got patched up.  There is of course a
chicken and egg problem here, because it's unclear to what degree the
commits drove the wrap date and to what degree the wrap date drove the
commits, but if we can't get fixes for those kinds of issues into the
tree on time, we are not going to release on time.  Now maybe if we
refuse to branch until we get these kinds of issues fixed, we'll fix
them sooner (or revert the patches that caused the problems in the
first place, which is another way to go).  That would be a positive
development.  But if we end up waiting just as long but with nothing
substantial getting committed in the meantime, that will be awful.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 2016-01-20 13:03:05 -0500, Tom Lane wrote:
> I am not sure what exactly ought to be different about them [small
> patches], but probably something should.  I think for small patches,
> we are using the CF app mostly to be sure things don't fall through
> the cracks, but maybe we don't need the whole process otherwise.

Aren't we already treating them differently on an ad-hoc basis? Several
committers fast-track such patches through the process. We could mark
them differently in the CF app, but I'm not sure what we'd want to
achieve by doing that. Such small patches are frequently enough
buggy. Committers can most of the time can easily polish them up, but
that still takes time; so I think tests/reviews of those are still
worthwhile.

So the only real difference I see an explicit difference would make is
making it easier to spot such patches when looking for an easy thing to
commit, and to make life for the commitfest manager easier. Not sure if
such a labeling would save more time than doing the labeling itself
would cost.

Andres

On 2016-01-20 13:13:29 -0500, Robert Haas wrote:
> (3) Andres's multixact reworks landed quite late and IMHO were not
> safe enough to back-patch, yet they needed to be in the release.  In
> my view, the last major fix for

At least in that case I was, for a long while, basically hoping for more
input, probably unwisely so. I think the 9.4 jsonb compressability thing
essentially was in a somewhat similar state for months; the proposed
patch was mostly there, but we didn't press ahead.  I think deadlines
and multiple people being blocked help tremendously with such issues.
They're often too complex for an individual and/or delayed due to lack
of concensus.


> But if we end up waiting just as long but with nothing substantial
> getting committed in the meantime, that will be awful.

That's true. Which I think means that we'll need to be agressive in
threatening reverts in such cases. That wouldn't have worked for 3), but
in the other two cases it'd have been doable.

Andres

On Wed, Jan 20, 2016 at 1:29 PM, Andres Freund <andres@anarazel.de> wrote:
> On 2016-01-20 13:13:29 -0500, Robert Haas wrote:
>> (3) Andres's multixact reworks landed quite late and IMHO were not
>> safe enough to back-patch, yet they needed to be in the release.  In
>> my view, the last major fix for
>
> At least in that case I was, for a long while, basically hoping for more
> input, probably unwisely so. I think the 9.4 jsonb compressability thing
> essentially was in a somewhat similar state for months; the proposed
> patch was mostly there, but we didn't press ahead.  I think deadlines
> and multiple people being blocked help tremendously with such issues.
> They're often too complex for an individual and/or delayed due to lack
> of concensus.

True.

>> But if we end up waiting just as long but with nothing substantial
>> getting committed in the meantime, that will be awful.
>
> That's true. Which I think means that we'll need to be agressive in
> threatening reverts in such cases. That wouldn't have worked for 3), but
> in the other two cases it'd have been doable.

Yes.  One problem is that it's very hard to get a patch reverted.  Tom
opined that too often patches roll along toward commit and that it's
too hard to say "no, we want that".  But that's nothing compared to
how hard it is to get a patch reverted.  It's true that, if Tom calls
for the revert, it more often than not happens, but nobody else can
really pull that off without a major war.

And, you know, I did my time fighting major wars to try to compress
the release schedule, and honestly, it wasn't that much fun.  The
process we have now is imperfect in many ways, but I no longer have
abuse heaped on me for wanting to boot a patch out of a CommitFest.
That may be bad for the project, but it's spared me a lot of grief
personally.  I think that many other people are in the same situation.
Everybody would like somebody else to be the schedule enforcer ...
unless they have something that they still want to get in, in which
case they would like nobody to do it.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 2016-01-20 18:53:54 +0000, Simon Riggs wrote:
> What is the point in having a special mailing list to discuss the release
> schedule plus a face-to-face dev meeting to discuss this if we are going to
> discuss it here?

That list exists to discuss concrete releases, and is separate so we
e.g. can mention there's security issues and such. This topic in
contrast quite validly merits input from more then the usual suspects
going to a developer meeting.

Andres

On Wed, Jan 20, 2016 at 11:13 AM, Andres Freund <andres@anarazel.de> wrote:
> That list exists to discuss concrete releases, and is separate so we
> e.g. can mention there's security issues and such. This topic in
> contrast quite validly merits input from more then the usual suspects
> going to a developer meeting.

I am not on the release list.


-- 
Peter Geoghegan

andres@anarazel.de (Andres Freund) writes:
> On 2016-01-20 18:53:54 +0000, Simon Riggs wrote:
>> What is the point in having a special mailing list to discuss the release
>> schedule plus a face-to-face dev meeting to discuss this if we are going to
>> discuss it here?

> That list exists to discuss concrete releases, and is separate so we
> e.g. can mention there's security issues and such. This topic in
> contrast quite validly merits input from more then the usual suspects
> going to a developer meeting.

In particular, we have several important people who won't be in Brussels
next week, so it's appropriate to hear their thoughts beforehand ...
        regards, tom lane

On Wed, Jan 20, 2016 at 2:26 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> andres@anarazel.de (Andres Freund) writes:
>> On 2016-01-20 18:53:54 +0000, Simon Riggs wrote:
>>> What is the point in having a special mailing list to discuss the release
>>> schedule plus a face-to-face dev meeting to discuss this if we are going to
>>> discuss it here?
>
>> That list exists to discuss concrete releases, and is separate so we
>> e.g. can mention there's security issues and such. This topic in
>> contrast quite validly merits input from more then the usual suspects
>> going to a developer meeting.
>
> In particular, we have several important people who won't be in Brussels
> next week, so it's appropriate to hear their thoughts beforehand ...

As somebody who definitely won't be there, +1 from me.  :-)

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 21/01/16 06:40, Tom Lane wrote:
> Magnus Hagander <magnus@hagander.net> writes:
>> That's pretty much what I suggested :)
>> Except that we need to do it for the last one as well. With the only
>> exception that the last one might be a bit longer. But the fact is that the
>> recent of CFs *and* releases, we've taken the approach of closing the CF
>> when everything in it is done or explicitly reviewed-and-bumped, and tried
>> really really hard not to bump things because nobody had time to look at
>> them. That's what I'm suggesting we change, and actually just cut them.
>> Yes, one of the reasons for the CFs was to allow people a fair chance to
>> get reviewed.But given that there isn't actually a deadline in practice
>> doesn't help with that.
> Yeah.  It's certainly unfair if someone's patch doesn't get reviewed,
> but there are only 24 hours in a day, and we have a limited pool of
> reviewer and committer manpower.  I think we just have to say that
> sometimes life is unfair.
>
> I think it might also be a good idea if we could somehow distinguish
> "nobody had time for that yet" from "nobody is interested", with an eye
> to flat-out rejecting patches that no one cares enough about to review.
> Maybe we could reduce the workload by inventing some kind of up-front
> filter whereby a patch isn't even a candidate to get reviewed unless, say,
> at least two people besides the author say "this sounds like it's worth
> pursuing".
>
> One of the other things I do not like about the current CF process is that
> it's created a default assumption that most submitted patches should get
> accepted eventually.  It is *very* hard to reject a patch altogether in
> the CF process: you more or less have to show cause why it should not get
> accepted.  That default is backwards.  Maybe this isn't the process' fault
> exactly, but it sure seems like that's how we approach patches now.
>
>             regards, tom lane
>
>
Possibly the emphasis should be on what patches should be ACCEPTED, 
rather than on what patches should be REJECTED?

Then there is less stigma in a patch missing out, and people don't have 
justify rejecting patches.


Cheers,
Gavin

On 01/20/2016 10:53 AM, Simon Riggs wrote:
> On 20 January 2016 at 15:40, Bruce Momjian <bruce@momjian.us
> <mailto:bruce@momjian.us>> wrote:
>
>     Many people where happy with our consistent releasing major releases in
>     September, e.g. 9.0 to 9.3:
>
>
> What is the point in having a special mailing list to discuss the
> release schedule plus a face-to-face dev meeting to discuss this if we
> are going to discuss it here?

Because, like core although they (may) have the final word, taking solid 
feedback from people not on that committee or having that discussion 
publicly lends itself to:
 * Community * Transparency * The Open Source way

>
> ISTM the wrong starting point to discuss plans in an unplanned way and
> assume that everyone has time to take part today, right now.

If it is important to them, they will make time. Everybody has a choice.

Sincerely,

JD

-- 
Command Prompt, Inc.                  http://the.postgres.company/                     +1-503-667-4564
PostgreSQL Centered full stack support, consulting and development.

On 1/20/16 1:44 PM, Robert Haas wrote:
> And, you know, I did my time fighting major wars to try to compress
> the release schedule, and honestly, it wasn't that much fun.  The
> process we have now is imperfect in many ways, but I no longer have
> abuse heaped on me for wanting to boot a patch out of a CommitFest.
> That may be bad for the project, but it's spared me a lot of grief
> personally.  I think that many other people are in the same situation.
> Everybody would like somebody else to be the schedule enforcer ...
> unless they have something that they still want to get in, in which
> case they would like nobody to do it.

Yeah, a lot of the ideas in this thread, while reasonable, are of the
sort "We need to be better about ..." which sounds a lot like "I need to
be better about exercising".  A system based purely on distributed
willpower isn't going to last very long, as we are finding out.

Sure, I'd like more than one party to review my patches, and I'd like to
spend more time doing additional reviews on other people's patches.  But
I can barely keep up as it is.  I know we need code reviews, but I think
it's never going to work well to the point that we are satisfied with
it.  Just look around the world, in software, open or commercial, or
even academics, and tell me who has peer reviews figured out.

My feeble attempts at alleviating this problem a bit are adding more
testing and removing outdated functionality, but both of those are also
incredibly abuse-prone activities.

FWIW, I do apologize for proposing the fifth commitfest (PGCon 2014).

Magnus Hagander wrote:
> On Wed, Jan 20, 2016 at 6:57 PM, Alvaro Herrera <alvherre@2ndquadrant.com>
> wrote:

> > Here's one area where the commitfest app could help the CFM.  I would
> > like to have a report that shows, for each person, the list of patches
> > where they are author, and the list of patches where they are reviewer.
> > Then I can check his reviewer score by skimming a couple of threads and
> > say "you don't have enough author credits, go away".
> >
> Report added. You will find a new "Reports" button at the bottom of the CF
> (as a CF admin) which has a link to it.

Excellent, this is very useful, thanks.

-- 
Álvaro Herrera                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On Wed, Jan 20, 2016 at 1:37 PM, Peter Eisentraut <peter_e@gmx.net> wrote:
> Yeah, a lot of the ideas in this thread, while reasonable, are of the
> sort "We need to be better about ..." which sounds a lot like "I need to
> be better about exercising".  A system based purely on distributed
> willpower isn't going to last very long, as we are finding out.

I have often thought that contributors have a tendency to be
unrealistic about how effective procedural changes can be in
alleviating our problems. We have always relied heavily on individual
acts of heroism. I expect no great changes there.

For example, you can't really expect a CF app SQL query to give you
anything greater than an extremely noisy idea of how much "review
goodwill" somebody has banked. My oldest pending patch in the CF app
is a boring refactoring patch that no one asked me to write. Should
that be held against me?

> My feeble attempts at alleviating this problem a bit are adding more
> testing and removing outdated functionality, but both of those are also
> incredibly abuse-prone activities.

I think that we've learned some lessons from the problems with 9.3. I
don't think that one of those lessons was "take more time to release".
There is reason to doubt that that would have changed matters one bit
with 9.3. It might be a good idea to formally state what those lessons
are.

-- 
Peter Geoghegan

On Wed, Jan 20, 2016 at 09:12:07AM -0800, Joshua Drake wrote:
> >I just don't buy the Ubuntu release model for our database.  Ubuntu is
> >trying to balance hot features vs stability, while we are really focused
> >on stability, similar to Debian.
> 
> I understand but I think we are missing out on an opportunity here.
> Notice that the shorter release cycle for STS will actually make
> some things easier. Including:
> 
>  * Increased test base (just like Fedora/Ubuntu)
>  * Increased early adopter testing (that is what early adopting is
> really about for us anyway)
>  * Decreased concerns about upgrades and ability to extend upgrade status.

I can see LTS working for plugin change, but not server binary changes.

--  Bruce Momjian  <bruce@momjian.us>        http://momjian.us EnterpriseDB
http://enterprisedb.com

+ As you are, so once was I. As I am, so you will be. +
+ Roman grave inscription                             +

On Wed, Jan 20, 2016 at 01:59:59PM -0800, Peter Geoghegan wrote:
> > My feeble attempts at alleviating this problem a bit are adding more
> > testing and removing outdated functionality, but both of those are also
> > incredibly abuse-prone activities.
> 
> I think that we've learned some lessons from the problems with 9.3. I
> don't think that one of those lessons was "take more time to release".
-------------------------
> There is reason to doubt that that would have changed matters one bit
> with 9.3. It might be a good idea to formally state what those lessons
> are.

I do think that is effectively a lesson we learned, wrong or not.

--  Bruce Momjian  <bruce@momjian.us>        http://momjian.us EnterpriseDB
http://enterprisedb.com

+ As you are, so once was I. As I am, so you will be. +
+ Roman grave inscription                             +

On Wed, Jan 20, 2016 at 07:48:07PM +0000, Simon Riggs wrote:
> If that is the case, then its too late to change.
> 
> March - last CF
> April - integration
> May - release Beta
> Sept - release Prod
> 
> It seems perfectly OK for me to have Beta start at beginning May and for us to
> release in September.
> 
> If the process takes more than 5 months (Apr - Sept) then there's something
> wrong with that process, rather than there being a problem with the schedule.

Well, consider the last commitfest finishes at the end of April, I don't
se how we could do integration and get to beta in May --- at least we
have never been able to do that.

--  Bruce Momjian  <bruce@momjian.us>        http://momjian.us EnterpriseDB
http://enterprisedb.com

+ As you are, so once was I. As I am, so you will be. +
+ Roman grave inscription                             +

On Wed, Jan 20, 2016 at 06:31:58PM +0100, Joel Jacobson wrote:
> On Wed, Jan 20, 2016 at 5:29 PM, Andres Freund <andres@anarazel.de> wrote:
> > I think one thing we should work on, is being absolutely religious about
> > requiring, say, 2 reviews for every nontrivial contribution.  We
> > currently seem to have a significantly increased submission rate, and at
> > the same time the number of reviews per patch has gone down
> > substantially.  I think the "honor" system has failed in that regard.
> 
> Good point, totally agree.
> 
> So the project should try to think of new ideas on how to incentivise reviewing?
> 
> I have three ideas so far:
> 
> 1. The way I see it, the honor system is based on being mentioned
> in the commit message and the release notes.
> 
> Authors are always mentioned in the release notes,
> but I see reviewers are mostly only mentioned in the commit messages.
> 
> Maybe more skilled developers would think it's cool to do reviewing
> if they were "paid" by also being mentioned in the release notes?
> 
> At least some skilled people would probably be motivated by it,
> in addition to the good feeling of doing something just because it's
> fun or important.

FYI, the fact that feature authors appear in the release notes is an
artifact of how we track who wrote which stuff, and is not designed for
rewarding, though it has that effect.  If we were to expand that to
cover reviewers, we would then be overburdinging that system and we
would probably end up removing all names from the release notes.

--  Bruce Momjian  <bruce@momjian.us>        http://momjian.us EnterpriseDB
http://enterprisedb.com

+ As you are, so once was I. As I am, so you will be. +
+ Roman grave inscription                             +

Bruce Momjian <bruce@momjian.us> writes:
> On Wed, Jan 20, 2016 at 01:59:59PM -0800, Peter Geoghegan wrote:
>> I think that we've learned some lessons from the problems with 9.3. I
>> don't think that one of those lessons was "take more time to release".
>                                             -------------------------
>> There is reason to doubt that that would have changed matters one bit
>> with 9.3. It might be a good idea to formally state what those lessons
>> are.

> I do think that is effectively a lesson we learned, wrong or not.

I think a lesson we *should* have learned by now is that we need to put
more emphasis on testing.  That includes not only spending more time on
it, but investing more in testing infrastructure.  The buildfarm has been
a huge advance in our ability to find/fix portability issues quickly, but
it does nothing to, say, assess crash safety.  Or identify performance
regressions.

As a concrete example, I recall that Heikki or someone had a tool for
checking WAL replay by comparing master and slave disk contents.  We
should make an effort to get that into a state where anyone can use it.
        regards, tom lane

On Wed, Jan 20, 2016 at 05:56:27PM -0500, Tom Lane wrote:
> Bruce Momjian <bruce@momjian.us> writes:
> > On Wed, Jan 20, 2016 at 01:59:59PM -0800, Peter Geoghegan wrote:
> >> I think that we've learned some lessons from the problems with 9.3. I
> >> don't think that one of those lessons was "take more time to release".
> >                                             -------------------------
> >> There is reason to doubt that that would have changed matters one bit
> >> with 9.3. It might be a good idea to formally state what those lessons
> >> are.
> 
> > I do think that is effectively a lesson we learned, wrong or not.
> 
> I think a lesson we *should* have learned by now is that we need to put
> more emphasis on testing.  That includes not only spending more time on
> it, but investing more in testing infrastructure.  The buildfarm has been
> a huge advance in our ability to find/fix portability issues quickly, but
> it does nothing to, say, assess crash safety.  Or identify performance
> regressions.

The two sobering blockers I can remember were multi-xact (9.3) and JSONB
compression (9.4).

--  Bruce Momjian  <bruce@momjian.us>        http://momjian.us EnterpriseDB
http://enterprisedb.com

+ As you are, so once was I. As I am, so you will be. +
+ Roman grave inscription                             +

On 2016-01-20 18:02:20 -0500, Bruce Momjian wrote:
> The two sobering blockers I can remember were multi-xact (9.3) and JSONB
> compression (9.4).

I don't see how those compare. The multixact stuff was all discovered a
fair bit after the release, the compression stuff pre release.

On Thu, Jan 21, 2016 at 12:05:19AM +0100, Andres Freund wrote:
> On 2016-01-20 18:02:20 -0500, Bruce Momjian wrote:
> > The two sobering blockers I can remember were multi-xact (9.3) and JSONB
> > compression (9.4).
> 
> I don't see how those compare. The multixact stuff was all discovered a
> fair bit after the release, the compression stuff pre release.

They were both cases where we were surprised and where hasty action did
or could have caused big problems.

--  Bruce Momjian  <bruce@momjian.us>        http://momjian.us EnterpriseDB
http://enterprisedb.com

+ As you are, so once was I. As I am, so you will be. +
+ Roman grave inscription                             +

On Wed, Jan 20, 2016 at 3:26 PM, Bruce Momjian <bruce@momjian.us> wrote:
> They were both cases where we were surprised and where hasty action did
> or could have caused big problems.

IMV MulitXacts were not a special case because of how many bugs there
were in 9.3.0. Rather, they were a special case because, to the
surprise of many, the number of bugs in each point release (or their
severity) did not really taper off. There was a particular qualitative
nature to the bugs. This seems to be because the design was
excessively complicated.

-- 
Peter Geoghegan

On Wed, Jan 20, 2016 at 2:56 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> I think a lesson we *should* have learned by now is that we need to put
> more emphasis on testing.  That includes not only spending more time on
> it, but investing more in testing infrastructure.  The buildfarm has been
> a huge advance in our ability to find/fix portability issues quickly, but
> it does nothing to, say, assess crash safety.  Or identify performance
> regressions.

I might add: Making testability an explicit goal of development from
the very beginning, during the design of the feature. In essence,
reducing the surface area for bugs when (not if) they happen, by being
able to easily simulate various stresses. Or better yet, making sure
that each mechanism modified has very little chance of interacting
with some other complicated mechanism in production. Things that
happen just once every 6 months on many production systems seemed to
be where MultiXacts kept giving trouble.

My pet example is how UPSERT doesn't change anything about the on-disk
representation of heap tuples, unless their xact aborted.

> As a concrete example, I recall that Heikki or someone had a tool for
> checking WAL replay by comparing master and slave disk contents.  We
> should make an effort to get that into a state where anyone can use it.

I think that that was initially Jeff Janes. Jeff also provided great
testing infrastructure for UPSERT, something that I was very grateful
for.

-- 
Peter Geoghegan

On Thu, Jan 21, 2016 at 8:51 AM, Peter Geoghegan <pg@heroku.com> wrote:
> On Wed, Jan 20, 2016 at 2:56 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> As a concrete example, I recall that Heikki or someone had a tool for
>> checking WAL replay by comparing master and slave disk contents.  We
>> should make an effort to get that into a state where anyone can use it.
>
> I think that that was initially Jeff Janes. Jeff also provided great
> testing infrastructure for UPSERT, something that I was very grateful
> for.

You mean the full-page write consistency tool? Heikki initiated that,
a common interface to be able to mask pages and make them consistent
for comparison on the master and the standby was also part of the
deal. Another tool that we had better IMO put some efforts in porting
into core is sqlsmith, which would actually be a complete rewrite
because the upstream code is under GPL license and depends on libpqxx.
-- 
Michael

On Thu, Jan 21, 2016 at 12:59 AM, Bruce Momjian <bruce@momjian.us> wrote:
> On Wed, Jan 20, 2016 at 10:55:07AM -0500, Robert Haas wrote:
>> On Wed, Jan 20, 2016 at 10:48 AM, Andres Freund <andres@anarazel.de> wrote:
>> > I think this has very little to do with commitfest schedules, and much
>> > more with the "early" forking of the new version branch. For both 9.4
>> > and 9.5 we essentially spent a couple months twiddling our thumbs.
>>
>> It's certainly true that we twiddled our thumbs quite a bit about
>> getting 9.5 ready to ship.  However, the old process where nobody
>> could get anything committed for six months out of the year blew
>> chunks, too.  Personally, I think that the solution is to cut off the
>> last CommitFest a lot sooner, and then reopen the tree for the next
>> release as soon as possible.  But this never works, because there are
>> always patches we want to slip in late.
>
> The bottom line is we can't sustain five commitfests and stay on time
> --- we need to go back to four, which I think is what we used to do.  We
> twiddled our thumbs back in the September-release years too, but had
> consistency because twiddling was built into the schedule.

Here I agree. Five commit fests is proving to be a lot and CF managers
are facing a severe burnout. Though I guess it sounded like a fine
idea at the moment this was begun (disclaimer: I was not there). So we
may want to do back to 4, and avoid too much overlapping between CFs
and what would be a stability period, without any commit fests going
on. It seems to me that the problem regarding the fact that fixes got
committed late is that committer's, author's and reviewer's attention
are getting distracted with commit fests and the new features proposed
there. Perhaps some people are more interested in implementing new
features than working on bugs and would just continue hacking and
arguing about new features, at least a stability period may attract
more committer attention into actual bug fixes, in short: no new
features can be committed until the previous versions has reached at
least beta2, rc, whatever. This may accelerate the stability process.
-- 
Michael

On Thu, Jan 21, 2016 at 1:32 PM, Michael Paquier
<michael.paquier@gmail.com> wrote:
> On Thu, Jan 21, 2016 at 12:59 AM, Bruce Momjian <bruce@momjian.us> wrote:
>> On Wed, Jan 20, 2016 at 10:55:07AM -0500, Robert Haas wrote:
>>> On Wed, Jan 20, 2016 at 10:48 AM, Andres Freund <andres@anarazel.de> wrote:
>>> > I think this has very little to do with commitfest schedules, and much
>>> > more with the "early" forking of the new version branch. For both 9.4
>>> > and 9.5 we essentially spent a couple months twiddling our thumbs.
>>>
>>> It's certainly true that we twiddled our thumbs quite a bit about
>>> getting 9.5 ready to ship.  However, the old process where nobody
>>> could get anything committed for six months out of the year blew
>>> chunks, too.  Personally, I think that the solution is to cut off the
>>> last CommitFest a lot sooner, and then reopen the tree for the next
>>> release as soon as possible.  But this never works, because there are
>>> always patches we want to slip in late.
>>
>> The bottom line is we can't sustain five commitfests and stay on time
>> --- we need to go back to four, which I think is what we used to do.  We
>> twiddled our thumbs back in the September-release years too, but had
>> consistency because twiddling was built into the schedule.
>
> Here I agree. Five commit fests is proving to be a lot and CF managers
> are facing a severe burnout. Though I guess it sounded like a fine
> idea at the moment this was begun (disclaimer: I was not there). So we
> may want to do back to 4, and avoid too much overlapping between CFs
> and what would be a stability period, without any commit fests going
> on. It seems to me that the problem regarding the fact that fixes got
> committed late is that committer's, author's and reviewer's attention
> are getting distracted with commit fests and the new features proposed
> there. Perhaps some people are more interested in implementing new
> features than working on bugs and would just continue hacking and
> arguing about new features, at least a stability period may attract
> more committer attention into actual bug fixes, in short: no new
> features can be committed until the previous versions has reached at
> least beta2, rc, whatever. This may accelerate the stability process.

Another idea popping to my mind in order to fix the CF manager burnout
and actually motivate people into becoming CF managers: say that one a
CF manager is done with a commit fest, folks on -hackers discuss if
the CFM has done a good job or not. If yes, he gets a travel paid to
the conference of his choice. That's assuming that there are actual
funds for that. Most people here perhaps may not care as their company
are usually willing to pay for such trip, but in some cases I am sure
that there are people who can be funded by their company *if* they do
a presentation to this conference. Just a random idea.
-- 
Michael

On Wed, Jan 20, 2016 at 8:32 PM, Michael Paquier
<michael.paquier@gmail.com> wrote:
> Perhaps some people are more interested in implementing new
> features than working on bugs and would just continue hacking and
> arguing about new features, at least a stability period may attract
> more committer attention into actual bug fixes, in short: no new
> features can be committed until the previous versions has reached at
> least beta2, rc, whatever. This may accelerate the stability process.

Or it might just accelerate the amount of time it takes to *declare*
having reached that milestone. Besides, do you really think that
people need to be able to commit something before seriously working on
it?

Rules that are expressly coercive are a *bad* idea.

-- 
Peter Geoghegan

On Wed, Jan 20, 2016 at 8:23 PM, Michael Paquier
<michael.paquier@gmail.com> wrote:
> Another tool that we had better IMO put some efforts in porting
> into core is sqlsmith, which would actually be a complete rewrite
> because the upstream code is under GPL license and depends on libpqxx.

Then Andreas Seltenreich better get a commit bit. I've seen him turn
around changes in sqlsmith to test new areas of Postgres in a couple
of days. That's a big reason why he has been so effective.

What benefit does porting sqlsmith for inclusion in core have? I can
only think of costs, including those that you mentioned.

-- 
Peter Geoghegan

On Thu, Jan 21, 2016 at 2:30 PM, Peter Geoghegan <pg@heroku.com> wrote:
> On Wed, Jan 20, 2016 at 8:23 PM, Michael Paquier
> <michael.paquier@gmail.com> wrote:
>> Another tool that we had better IMO put some efforts in porting
>> into core is sqlsmith, which would actually be a complete rewrite
>> because the upstream code is under GPL license and depends on libpqxx.
>
> Then Andreas Seltenreich better get a commit bit. I've seen him turn
> around changes in sqlsmith to test new areas of Postgres in a couple
> of days. That's a big reason why he has been so effective.
>
> What benefit does porting sqlsmith for inclusion in core have? I can
> only think of costs, including those that you mentioned.

We have automatic buildfarm coverage on many platforms. Perhaps we
could live without that with a buildfarm module though.
-- 
Michael

On Wed, Jan 20, 2016 at 9:37 PM, Michael Paquier
<michael.paquier@gmail.com> wrote:
> We have automatic buildfarm coverage on many platforms. Perhaps we
> could live without that with a buildfarm module though.

It's not clear that buildfarm coverage makes sense for sqlsmith. If it
does make sense, I see no reason why introducing it should be attached
to sqlsmith's inclusion in core.

-- 
Peter Geoghegan

Michael Paquier <michael.paquier@gmail.com> writes:
> On Thu, Jan 21, 2016 at 2:30 PM, Peter Geoghegan <pg@heroku.com> wrote:
>> What benefit does porting sqlsmith for inclusion in core have? I can
>> only think of costs, including those that you mentioned.

> We have automatic buildfarm coverage on many platforms. Perhaps we
> could live without that with a buildfarm module though.

I do not think we should necessarily try to include every testing tool
in the core distribution.  What is important is that they be readily
available: easy to find, easy to use, documented, portable.  "Same
license as the PG core code" is not on that list.

An immediately relevant example is that the buildfarm server and client
code aren't in the core distribution, and AFAIR no one has suggested
that they need to be.
        regards, tom lane

On Wed, Jan 20, 2016 at 12:40:04PM -0500, Tom Lane wrote:
> I think it might also be a good idea if we could somehow distinguish
> "nobody had time for that yet" from "nobody is interested", with an eye
> to flat-out rejecting patches that no one cares enough about to review.
> Maybe we could reduce the workload by inventing some kind of up-front
> filter whereby a patch isn't even a candidate to get reviewed unless, say,
> at least two people besides the author say "this sounds like it's worth
> pursuing".
> 
> One of the other things I do not like about the current CF process is that
> it's created a default assumption that most submitted patches should get
> accepted eventually.  It is *very* hard to reject a patch altogether in
> the CF process: you more or less have to show cause why it should not get
> accepted.  That default is backwards.  Maybe this isn't the process' fault
> exactly, but it sure seems like that's how we approach patches now.

These paragraphs point to different facets of the same problem: community
conventions make rejecting a patch a substantial project in its own right.  +1
for experimenting with one or more ways to expedite that.  There's your
two-ACKs idea above, and we floated[1] a few more at the 2015-06-16 meeting:
 This was followed by discussion of how we arrange the CFs. Noah suggested a prioritization system. He suggested a
systemwhere various committers can provide feedback on stuff as triage. Simon agreed and volunteered. Haas suggested a
2-committerveto. Frost said we need a formal way to "nack" something. We need to triage stuff earlier in the process.
Haassays the big problem is the endless arguments because we don't want to say "no" to people because we have scarce
committertime.
 

I recommend trying the two-ACKs plan for one CF.  Keep it if it helps
markedly; otherwise, try another of these variants for the CF after that one.

Thanks,
nm

[1] https://wiki.postgresql.org/wiki/PgCon_2015_Developer_Meeting#9.6_Schedule

On Wed, Jan 20, 2016 at 06:58:24PM +0000, Simon Riggs wrote:
> The main problem is the length of the integration phase, which is mostly
> where nothing happens.

The open items wiki page saw steady change from 30 April to 28 December[1];
the two longest quiet periods were an eleven-day gap from 21 August and a
nine-day gap from 16 December.  Nineteen distinct contributors were finding,
fixing or otherwise editing data about 9.5 defects.  The integration work is
too slow, yes, but this persistent myth of a dead period is false.

For my part, I spent 1 July to 11 December reviewing 9.5 commits and reporting
or fixing defective 9.5 work.  (I did spoil myself with a few breaks fixing
pre-9.5 bugs.)  I doubt the 9.5 new bugs would have run dry within the next
five months, either, but the release team picked a fair juncture to give up
and call it dot-zero.

nm

[1] https://wiki.postgresql.org/index.php?title=PostgreSQL_9.5_Open_Items&limit=500&action=history

On Wed, Jan 20, 2016 at 11:46 PM, Peter Geoghegan <pg@heroku.com> wrote:
> On Wed, Jan 20, 2016 at 8:32 PM, Michael Paquier
> <michael.paquier@gmail.com> wrote:
>> Perhaps some people are more interested in implementing new
>> features than working on bugs and would just continue hacking and
>> arguing about new features, at least a stability period may attract
>> more committer attention into actual bug fixes, in short: no new
>> features can be committed until the previous versions has reached at
>> least beta2, rc, whatever. This may accelerate the stability process.
>
> Or it might just accelerate the amount of time it takes to *declare*
> having reached that milestone.

Yes. I think that's been a problem already, and closing development
for prolonged periods of time will make it worse.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Bruce Momjian wrote:

> FYI, the fact that feature authors appear in the release notes is an
> artifact of how we track who wrote which stuff, and is not designed for
> rewarding, though it has that effect.

I think you can claim this all you want, but I'd be surprised if anyone
actually believed it.  You're saying that we do this so that we can
properly attribute code to developers so that we can pester the right
guy when things go wrong; but in reality when we investigate a bug we
never look at the release notes; what we do is look at the Git logs,
which already contain this information.

To the public at large I'm pretty sure having the names there is a
reward for having developed stuff, not a blame sign.  Do real users go
back to 9.3 to see that "oh this Alverro dude wrote multixacts which had
so many bugs, we should really hang him by the bollocks"?  I don't think
so.  People like Robert, Andres, Noah are more likely to want me hung
because they worked so hard to get the bugs fixed.

> If we were to expand that to
> cover reviewers, we would then be overburdinging that system and we
> would probably end up removing all names from the release notes.

To me, this reads just as a threat that "if you disturb the waters here
I will just remove all names and you will all be the poorer for it."
Please don't do that, it looks rude.

I think it is a disservice to the project to deny the reviewers the bit
of coolness factor derived from having their names in the release notes
-- or maybe just *somewhere*.  If you want to remove the developer names
from the release notes to avoid bloating them, perhaps what we can do is
put names of authors and reviewers in a page in the website (probably
not the wiki, though, because it's easy to vandalize that and many
people may have concerns regarding the legitimacy of anyone listed
there.)

-- 
Álvaro Herrera                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On Fri, Jan 22, 2016 at 10:43 AM, Alvaro Herrera
<alvherre@2ndquadrant.com> wrote:
>> If we were to expand that to
>> cover reviewers, we would then be overburdinging that system and we
>> would probably end up removing all names from the release notes.
>
> To me, this reads just as a threat that "if you disturb the waters here
> I will just remove all names and you will all be the poorer for it."
> Please don't do that, it looks rude.

For my part, I am not sure the names in the release notes are actually
all that helpful.  What does and does not get credited there and how
it gets credited is more random than I would like.  And it's often
hard to draw the line as to who contributed enough to deserve credit
and who did not.  In commit messages you can try to spell out the
details, but as Bruce says, there's no room for that in the release
notes.  If a feature is credited as (Robert Haas, X, Y, Z) it's clear
that I'm the principle author, but it's unclear whether I did 40% of
the work or 90% of the work.  If it's credited as (X, Robert Haas),
that could mean X came up with the idea and wrote a patch which I then
completely re-wrote from scratch using a different design and
committed; or it could mean X came with an idea that was basically
good and I wrote and debugged a 1000-line patch, and I rewrote 10% of
it for some reason before committing.  So even today, it's pretty hard
to use the release notes as a way to measure depth of contribution.
The problem is worse for people most of whose work is reviewing and
committing rather than authoring, but if you put everybody's name on
everything they reviewed, that wouldn't fix it.  Instead, you'd tend
to overvalue people who do a lot of shallow reviews relative to people
who do a few very deep ones.  I think we just have to resign ourselves
to the fact that judging depth of contribution from the release notes,
or even the commit log, is going to be rather difficult; and that
different people will have different opinions about whose work is most
meritorious.

In other words, I fear that putting names in the release notes is just
politicizing a process that ought to be about presenting the
community's work in the best way possible.  A new release ought to be
about putting the whole community forward in the best light.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 1/21/16 2:29 AM, Amit Kapila wrote:
> I also think there should be some way to give credit to CFM, if it is
> difficult to do anything related to money, then we can enforce that if
> CFM submits any patches for next CF, then those should be prioritised.

Personally, I don't see why we have our scarcest resource doing what is 
essentially a project management task, especially when at least one 
commercial company has offered to donate paid staff time.

I don't think the last CF of 9.6 is the time to experiment, but I think 
we should try using a PM for the first CF of 9.7.
-- 
Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX
Experts in Analytics, Data Architecture and PostgreSQL
Data in Trouble? Get it in Treble! http://BlueTreble.com

On 1/20/16 11:49 PM, Tom Lane wrote:
> Michael Paquier <michael.paquier@gmail.com> writes:
>> On Thu, Jan 21, 2016 at 2:30 PM, Peter Geoghegan <pg@heroku.com> wrote:
>>> What benefit does porting sqlsmith for inclusion in core have? I can
>>> only think of costs, including those that you mentioned.
>
>> We have automatic buildfarm coverage on many platforms. Perhaps we
>> could live without that with a buildfarm module though.
>
> I do not think we should necessarily try to include every testing tool
> in the core distribution.  What is important is that they be readily
> available: easy to find, easy to use, documented, portable.  "Same
> license as the PG core code" is not on that list.
>
> An immediately relevant example is that the buildfarm server and client
> code aren't in the core distribution, and AFAIR no one has suggested
> that they need to be.

Right. What I think would be far more useful is making it easier to 
explicitly test things (better tools + design for test), and something 
akin to buildfarm that will run automated testing on submitted patches.

Put another way: it's stupid that we even ask reviewers to waste time 
running make check. That can be automated. Ideally reviewers shouldn't 
be doing any testing, because the tests that are part of the patch 
should answer every question they would have, but I don't see that 
happening until we have a separate automation-only target that we don't 
care how long it takes to run.
-- 
Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX
Experts in Analytics, Data Architecture and PostgreSQL
Data in Trouble? Get it in Treble! http://BlueTreble.com

On 1/20/16 11:40 AM, Tom Lane wrote:
> Yeah.  It's certainly unfair if someone's patch doesn't get reviewed,
> but there are only 24 hours in a day, and we have a limited pool of
> reviewer and committer manpower.  I think we just have to say that
> sometimes life is unfair.

I think that's a great way to ensure we shrink the pool of reviewers 
when someone works on a patch and then it goes nowhere. I find it rather 
difficult to get feedback on ideas before I spend the time to code 
something, it's got to be even worse for someone the community doesn't 
know. So if we're going to do this, I think there must be a mechanism 
for a patch idea/design to be approved.

I think we also need to be careful about -hackers being the only place 
feature desirability is measured. There's an entire world of users out 
there that aren't even on -general. If some feature doesn't really 
interest -hackers but there's 50 users that want it and someone willing 
to work on it, ISTM we should make efforts to get it committed.
-- 
Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX
Experts in Analytics, Data Architecture and PostgreSQL
Data in Trouble? Get it in Treble! http://BlueTreble.com

On 1/20/16 4:29 PM, Bruce Momjian wrote:
> On Wed, Jan 20, 2016 at 09:12:07AM -0800, Joshua Drake wrote:
>>> I just don't buy the Ubuntu release model for our database.  Ubuntu is
>>> trying to balance hot features vs stability, while we are really focused
>>> on stability, similar to Debian.
>>
>> I understand but I think we are missing out on an opportunity here.
>> Notice that the shorter release cycle for STS will actually make
>> some things easier. Including:
>>
>>   * Increased test base (just like Fedora/Ubuntu)
>>   * Increased early adopter testing (that is what early adopting is
>> really about for us anyway)
>>   * Decreased concerns about upgrades and ability to extend upgrade status.
>
> I can see LTS working for plugin change, but not server binary changes.

s/LTS/STS/?

In any case, I think JD is onto something here. As someone that focuses 
more on user experience than "deep core" code, I already find yearly 
releases to be quite inconvenient. It's hard to find the motivation to 
make a minor improvement in something (especially knowing how hard it 
will be to get the patch approved) knowing that it won't see the light 
of day for a year, and realistically I won't be able to use it with any 
clients that are in production for 2-3 years.

Given the high level of extensibility that we have, maybe it would be 
good to logically segregate stuff into things that are deeply embedded 
in the "core" code (ie: on-disk format) from things that are much easier 
to change when necessary (like add-on functions or PLs). Things like new 
JSON operators could be released much more rapidly that way.
-- 
Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX
Experts in Analytics, Data Architecture and PostgreSQL
Data in Trouble? Get it in Treble! http://BlueTreble.com

On 2016-01-22 08:40:28 -0600, Jim Nasby wrote:
> Ideally reviewers shouldn't be doing any testing, because the tests
> that are part of the patch should answer every question they would
> have, but I don't see that happening until we have a separate
> automation-only target that we don't care how long it takes to run.

I think that's completely wrong.

Yes, more tests are good, and we need a place for longer running
tests. But assuming that every patch author will create a testsuite that
covers every angle is just about akin to assuming every submitter will
deliver perfect, bug free code. And we know how well that turns out.

I think actively trying to break a feature, and postgres in general, is
one of the most important tasks of reviewers and testers. And with that
I don't mean trying to run "make check". Look e.g. at the tests Jeff
Janes has performed, what the recent plug tests of Tomas Vondra brought
to light, or at what the full page write checker tool of Heikki's
showed.

Andres

On 2016-01-22 08:18:45 -0600, Jim Nasby wrote:
> Personally, I don't see why we have our scarcest resource doing what is
> essentially a project management task, especially when at least one
> commercial company has offered to donate paid staff time.

Because so far all CFs that weren't managed by somebody involved on the
code level worked even less than the rest. You need to be able to judge
how complex and ready a patch is to some degree. You need to know who to
prod for a specific patch.

Yes, a PM can learn to do that. But it's not something a freshly hired
person can do. You need a fair bit of community involvement.

Andres

On 2016-01-22 08:50:15 -0600, Jim Nasby wrote:
> I think that's a great way to ensure we shrink the pool of reviewers when
> someone works on a patch and then it goes nowhere.

True, it really sucks. But what's your counter proposal? Commitfests
dragging on forever, and people burning out on continually feeling they
need to work on them? Hasn't worked out well.

On Fri, Jan 22, 2016 at 6:21 PM, Andres Freund <andres@anarazel.de> wrote:
> On 2016-01-22 08:50:15 -0600, Jim Nasby wrote:
>> I think that's a great way to ensure we shrink the pool of reviewers when
>> someone works on a patch and then it goes nowhere.
>
> True, it really sucks. But what's your counter proposal? Commitfests
> dragging on forever, and people burning out on continually feeling they
> need to work on them? Hasn't worked out well.

Well my point of view is that the entire reason for commitfests is to
ensure every patch gets at least some feedback. If we're going to
bounce patches without any feedback and only work on the patches that
catchh people's interest then that's back where we were before
commitfests. We used to see a lot of patches sit basically ignored
until release time when Tom or someone would pick them up and rewrite
them from scratch because that would be faster than trying to explain
what's wrong and waiting for the original author to rewrite it. It
really sucks for the original author to be ignored for a year and I
think it's how we lost a lot of potential contributors.

I think it's true that there are basically two successful lanes
patches can be in. a) They're of wide interest and need concerted
ongoing effort by multiple people to proceed and b) they're of narrow
interest but really just need a thumbs up or pointer to what direction
to head and the original author can proceed. The first clogs up the
commitfest and dominates people's time when it probably belongs more
in the normal development process. The latter is really what they were
designed for.

Perhaps it would make sense to alternate between "commitfests" that
are intended to return feedback to authors so they can work on their
patch and "developfests" where larger patches that need more
discussion get covered. The latter are expected to drag on but
shouldn't block other work whereas the commitfests are expected to get
relatively short reviews and be bounced as "returned with feedback"
quickly.

Or perhaps we should just have two different "returned with feedback",
one of which is "under discussion" which means the patch has seen some
discussion and therefore doesn't need to be in the commitfest any more
regardless of whether it actually resolved the issues authoritatively.
Making decisions in a consensus-driven community is just hard and we
could use some lessons in how to say no or how to resolve
irreconcilable conflicts but barring solving those issues it would at
least be nice to remove them from the critical path blocking other
patches and making the process feel interminable for bystanders.

-- 
greg

On 20.01.2016 22:37, Peter Eisentraut wrote:
> On 1/20/16 1:44 PM, Robert Haas wrote:
>> And, you know, I did my time fighting major wars to try to compress
>> the release schedule, and honestly, it wasn't that much fun.  The
>> process we have now is imperfect in many ways, but I no longer have
>> abuse heaped on me for wanting to boot a patch out of a CommitFest.
>> That may be bad for the project, but it's spared me a lot of grief
>> personally.  I think that many other people are in the same situation.
>> Everybody would like somebody else to be the schedule enforcer ...
>> unless they have something that they still want to get in, in which
>> case they would like nobody to do it.
>
> Yeah, a lot of the ideas in this thread, while reasonable, are of the
> sort "We need to be better about ..." which sounds a lot like "I need to
> be better about exercising".  A system based purely on distributed
> willpower isn't going to last very long, as we are finding out.
>
> Sure, I'd like more than one party to review my patches, and I'd like to
> spend more time doing additional reviews on other people's patches.  But
> I can barely keep up as it is.  I know we need code reviews, but I think
> it's never going to work well to the point that we are satisfied with
> it.  Just look around the world, in software, open or commercial, or
> even academics, and tell me who has peer reviews figured out.

Nobody, but there are different solutions. And the same solutions works 
different in quality and quantity in the different projects.
In FreeBSD for example there is an online tool for review 
(http://review.freebsd.org) which was opened to public. There you can 
review any code, left comments in the parts you wanted, submit different 
users to it etc.
It is not perfect, but a huge step forward for the project. And 
something i misses here often.
But as stated earlier in another thread: for a not-so-deep-involved 
volunteer, it is often unclear *what* to review. The threads are long 
and often there is no final description about how the patch is supposed 
to work. That make testing quite hard and time consuming.

It is not just about the schedule, the reviewer and participants. Its 
also some more basic.

Greetings,
Torsten

On Wed, Jan 20, 2016 at 01:59:59PM -0800, Peter Geoghegan wrote:
> I think that we've learned some lessons from the problems with 9.3. I
> don't think that one of those lessons was "take more time to release".
> There is reason to doubt that that would have changed matters one bit
> with 9.3. It might be a good idea to formally state what those lessons
> are.

Hindsight of 9.3 taught me to monitor three additional[1] signals for every
commit.  Each is weak individually, firing often on harmless commits.
However, if commit 0ac5ad5 landed again today, I would at least prioritize
further analysis based on all three signals being present:


1. Patch came together shortly before a deadline.

The tree sees a bunch of feature commits just before feature freeze.  This
list sees many fresh patch versions just before the last CommitFest submission
deadline.  For an important subset, time pressure led the author or committer
to reduce vigilance.  This is an analog signal, and I write "came together" to
be deliberately vague.  The more the various steps of development work
(design, implementation, review, commit) cluster near some deadline, the more
to worry.


2. Committer disclaimed knowledge of the patch's correctness.

From commit 0ac5ad5 procedural history:
> Tom Lane said at one point "this is too complex to maintain".  Several
> times during the development I feared he might well be right.  I am sure
> he will be discovering many oversights and poor design choices, when
> gets around to reviewing the code; hopefully some extra effort will be
> all that's needed to make the whole thing work sanely and not eat
> anyone's data.  I just hope that nothing so serious comes up that the
> patch will need to be reverted completely.

Author/committer personality is a large factor in determining whether this
signal arrives.  If I do see it, I now believe the author about the patch
being dangerous.


3. Committer == author and no other committer certified the patch.

By "certify", I mean a direct statement like "This patch is correct." or "This
feature is now bug-free." etc.  Posting a review does not, by itself, qualify,
and reviewers regularly do enough to earn a co-author credit without having
certified any version of the patch.  Being listed as the patch's git
author/committer _does_ imply certification.

This signal is common, particularly for smaller patches, and it ought to
remain common.  The product would advance slowly if every contrib module or
vacuumdb flag took full attention from two committers.  Every release has a
few patches that do deserve this level of attention, though, and they rarely
attract it.  Index-only scans did; commit a2822fb had a non-author committer
despite an author having a commit bit.  Committers should recognize particular
self-authored work as too grave to self-certify, submit it for review, and not
commit it until and unless another committer certifies it.  Indeed, I regard
that recognition of one's own limits as the central qualification for being a
committer.  I have failed at it myself, though.


Watching those signals is the lesson I took from the PostgreSQL 9.3 aftermath.

Thanks,
nm

[1] They joined older signals, some too laborious to measure for every commit:
- complexity of the area(s) patched
- plausible scope of breakage from defects, given the area(s) patched
- gravity of follow-on repair commits

On Fri, Jan 22, 2016 at 1:16 PM, Simon Riggs <simon@2ndquadrant.com> wrote:
> It has one important effect of current interest: establishing the truth that
> multiple people and multiple companies are involved in producing and
> maintaining PostgreSQL. Whether the names are properly attributed will
> always be a time-consuming task, but I will oppose any attempt to remove or
> obscure evidence of who develops PostgreSQL, wherever that occurs.

Obscuring evidence of who develops PostgreSQL was not my intention.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Mon, Jan 25, 2016 at 9:51 AM, Noah Misch <noah@leadboat.com> wrote:
> This signal is common, particularly for smaller patches, and it ought to
> remain common.  The product would advance slowly if every contrib module or
> vacuumdb flag took full attention from two committers.  Every release has a
> few patches that do deserve this level of attention, though, and they rarely
> attract it.  Index-only scans did; commit a2822fb had a non-author committer
> despite an author having a commit bit.

Yeah, I was a bit surprised when Tom committed that, but I was
pleased.  I didn't have huge confidence in it at that point, and
having Tom come along and give a hand was a great relief.  I think he
not only committed the base patch but improved a very healthy number
of things about it as well.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 1/25/16 2:48 AM, Torsten Zühlsdorff wrote:
> Nobody, but there are different solutions. And the same solutions works
> different in quality and quantity in the different projects.
> In FreeBSD for example there is an online tool for review
> (http://review.freebsd.org) which was opened to public. There you can
> review any code, left comments in the parts you wanted, submit different
> users to it etc.
> It is not perfect, but a huge step forward for the project. And
> something i misses here often.
> But as stated earlier in another thread: for a not-so-deep-involved
> volunteer, it is often unclear *what* to review. The threads are long
> and often there is no final description about how the patch is supposed
> to work. That make testing quite hard and time consuming.

I agree better code review tooling could help a bit.  The URL you post
above doesn't work at the moment (for me), though.

All,

So the proximate cause of late releases are the following:

1. patches are getting kicked down the road from one CF to another, creating a big pileup in the final CF.  This is
exactlylike the huge pile of unreviewed patches we used to have before the CF system.
 

2. Once the last CF is closed, everyone goes off and starts working on the next version, leaving a few people to handle
gettingthe production release integrated and debugged.  This is partly because nobody likes doing that work, and partly
becausemost hackers aren't clear how they can help.
 

So, I have a modest suggestion on how to change all this:

Let's do quarterly development releases and supported production releases every 18 months.

A 3-month release cycle would let people see their code go into the wild a lot faster; basically we'd do a CF then a
developmentrelease.  That would limit pileup issues around people losing interest/moving on, forgetting what was going
on,and conflicts between various features in development.  A shorter release/dev cycle is more manageable. By having
supportedproduction releases 50% less often, we could keep the overall number of versions we need to patch the same as
itis now.
 

The alternative to this is an aggressive recruitment and mentorship program to create more major contributors who can
dodeep review of patches.  But that doesn't seem to have happened in the last 5 years, and even if we started it now,
itwould be 2 years before it paid off.
 

-- 
Josh Berkus
Red Hat OSAS
(opinions are my own)

On 01/26/2016 02:09 AM, Peter Eisentraut wrote:> On 1/25/16 2:48 AM, Torsten Zühlsdorff wrote:
>> In FreeBSD for example there is an online tool for review>> (http://review.freebsd.org) which was opened to public.
Thereyou can>> review any code, left comments in the parts you wanted, submit different>> users to it etc. 
> I agree better code review tooling could help a bit.  The URL you post> above doesn't work at the moment (for me),
though.

That's a typo, it's reviews.freebsd.org.

Hi,

On 2016-01-26 00:26:18 -0600, Joshua Berkus wrote:
> Let's do quarterly development releases and supported production
> releases every 18 months.

> A 3-month release cycle would let people see their code go into the wild a lot faster; basically we'd do a CF then a
developmentrelease.
 

There are reason for/against doing that, but why would it actually
change the problem significantly? It won't change the amount of
necessary review, debugging and bugfixing by a very significant
amount. One CF / 3 months might actually make it harder to get a patch
into a revieawable state, because the time-to-feedback is larger.


> The alternative to this is an aggressive recruitment and mentorship
> program to create more major contributors who can do deep review of
> patches.  But that doesn't seem to have happened in the last 5 years,
> and even if we started it now, it would be 2 years before it paid off.

I think the amount of review and maintenance time is the largest part of
the problem here, and is mostly unaffected by the manner we actually
schedule releases.  The discussions around dropping patches on the floor
are partially a question of that, and partially a question of not
acceppting to maintain things that are of low interest.

But I don't really see "aggressive recruitment and mentorship" really
fixing this, even though there are other good reasons to work more on
that side.  I think more fundamentally the issue is that that doing a
"deep" review of a bigger patches takes so much time and knowledge, that
it's not realistic to do so in ones own time anymore. You can if you're
very dedicated over a long time, but realistically in most cases it
requires your employer having an interest in you doing that.  Quite
obviously there's an increasing number of employers paying people to
submit things to postgres, whereas there seems no corresponding uptick
on the review side.  Unless the "paying people to do review side" is
solved (which has a *long* lead time), I don't see more recruiting,
different CF schedules, or anything like that really fixing the critical
bottlenecks.  Reducing the amount of work, i.e. dropping patches not
interesting or too complex, seems to be the only solution until then.


As a short aside: I think another side of the coin is that, if you're
dedicated to work on a cool open source project, these days you'll find
a paying open source job so quickly, that a lot of people that would
have worked on e.g. postgres in their evenings, now just hack something
they're paid both during the day and the evening.


Andres

On 26.01.2016 02:09, Peter Eisentraut wrote:
> On 1/25/16 2:48 AM, Torsten Zühlsdorff wrote:
>> Nobody, but there are different solutions. And the same solutions works
>> different in quality and quantity in the different projects.
>> In FreeBSD for example there is an online tool for review
>> (http://review.freebsd.org) which was opened to public. There you can
>> review any code, left comments in the parts you wanted, submit different
>> users to it etc.
>> It is not perfect, but a huge step forward for the project. And
>> something i misses here often.
>> But as stated earlier in another thread: for a not-so-deep-involved
>> volunteer, it is often unclear *what* to review. The threads are long
>> and often there is no final description about how the patch is supposed
>> to work. That make testing quite hard and time consuming.
>
> I agree better code review tooling could help a bit.  The URL you post
> above doesn't work at the moment (for me), though.

I'm sorry, the url contains a typo. The correct one is:
https://reviews.freebsd.org/

Greetings,
Torsten

On 1/22/16 12:14 PM, Andres Freund wrote:
> On 2016-01-22 08:40:28 -0600, Jim Nasby wrote:
>> Ideally reviewers shouldn't be doing any testing, because the tests
>> that are part of the patch should answer every question they would
>> have, but I don't see that happening until we have a separate
>> automation-only target that we don't care how long it takes to run.
>
> I think that's completely wrong.
>
> Yes, more tests are good, and we need a place for longer running
> tests. But assuming that every patch author will create a testsuite that
> covers every angle is just about akin to assuming every submitter will
> deliver perfect, bug free code. And we know how well that turns out.
>
> I think actively trying to break a feature, and postgres in general, is
> one of the most important tasks of reviewers and testers. And with that
> I don't mean trying to run "make check". Look e.g. at the tests Jeff
> Janes has performed, what the recent plug tests of Tomas Vondra brought
> to light, or at what the full page write checker tool of Heikki's
> showed.

IIRC Jeff's tests are scripted and obviously the page write checker is 
as well. I don't recall the exact methodology Tomas was using but I 
suspect it could also be scripted if you had a way to pull the plug via 
software (via a power management unit or maybe kill -9 of a VM). All of 
that is stuff that can and should be automated. Presumably it won't ever 
be part of the Makefile tests, but that's fine. Heck, the test scripts 
could stay in completely separate repos.

Where the code lives isn't the issue; it's getting stuff like this 
automated so humans can go back do doing things that can't be automated.
-- 
Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX
Experts in Analytics, Data Architecture and PostgreSQL
Data in Trouble? Get it in Treble! http://BlueTreble.com