Thread: 9.1 Beta
Judging by the number of new threads about development for 9.2, I think its time we declared 9.1 Beta. I just had a conversation with some Debian developers about how PostgreSQL 9.0 got pulled out of their release because we delayed by 3 weeks. So we missed our slot to deliver useful new features to our very best supporters by 2 years. I really hope that was deliberate. I've never understood why we timebox useful development, yet tweaking is allowed to go on without limit. Personally, I don't see the rationale to allow developers some kind of priority over their input. This tweaking period is essentially a time when insiders can put their votes in, but nobody else can. Beta is where we get feedback from a wider audience. The sooner we declare Beta, the sooner people will test. Then we will have user feedback, bugs to fix etc.. Everybody is very clearly sitting idle. With a longer bug list we will make faster progress to release. We're just wasting time. If we had a hard date for feature freeze, lets have a hard date for Beta of +2 months (next time), and +2.5 months now. (I know +1 month was suggested, well that's just unrealistic). Beta is a great time to resolve difficult decisions, by opening the floor to wider debate and feedback. Delaying beta because we still have unresolved issues is exactly backwards of what we should be doing. Let's hear from a wider audience. Vox populi, vox dei -- Simon Riggs http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
Simon Riggs <simon@2ndQuadrant.com> writes:
> Judging by the number of new threads about development for 9.2, I
> think its time we declared 9.1 Beta. I just had a conversation with
> some Debian developers about how PostgreSQL 9.0 got pulled out of
> their release because we delayed by 3 weeks. So we missed our slot to
> deliver useful new features to our very best supporters by 2 years. I
> really hope that was deliberate.
We do *not* make release decisions based on Debian's schedule. Even if
we wanted to, going beta is hardly likely to affect their decisions.
The correct question is whether we're ready for beta, and I would say
the answer is clearly no, unless you have a pretty low standard for what
"ready for beta" means. Perhaps it would be suitable to discuss what
the standard for that really ought to be; but I don't agree in the
slightest that we ought to decide based on predetermined calendar dates
rather than the state of the code.
> If we had a hard date for feature freeze, lets have a hard date for
> Beta of +2 months (next time), and +2.5 months now. (I know +1 month
> was suggested, well that's just unrealistic). Beta is a great time to
> resolve difficult decisions, by opening the floor to wider debate and
> feedback.
The reason we get wider testing during beta is that people have some
confidence (perhaps misplaced) that the database won't eat their data.
Releasing alpha-grade code and calling it beta isn't going to get us
wider testing ... at least, not more than once.
regards, tom lane
On Fri, Mar 25, 2011 at 6:18 PM, Simon Riggs <simon@2ndquadrant.com> wrote: > I've never understood why we timebox useful development, yet tweaking > is allowed to go on without limit. Personally, I don't see the > rationale to allow developers some kind of priority over their input. > This tweaking period is essentially a time when insiders can put their > votes in, but nobody else can. Beta is where we get feedback from a > wider audience. I think 9.0 got delayed quite a bit by the fact that we need approximately 347 people to wrap a release, and they all had vacations at different times over the summer. The code was pretty stable by July 1; I think we could easily have released in August if we had a slightly less awkward process for getting these things out the door. > The sooner we declare Beta, the sooner people will test. Then we will > have user feedback, bugs to fix etc.. Everybody is very clearly > sitting idle. With a longer bug list we will make faster progress to > release. We're just wasting time. I can't resist observing that if you want beta to happen sooner, it would be better not to commit major and largely unreviewed patches three weeks after the end of the last CommitFest. Before you insist that it was reviewed, the version that was actually committed bore so little resemblance to the versions that were posted earlier that any earlier review that was done was basically meaningless in terms of ensuring that the final product was bug free, and it wasn't and isn't.I complained *repeatedly* about the need to get bothcollation support and sync rep finished and committed sooner, for exactly this reason. We are now reaping the entirely predictable fruit of having failed to make that happen. But for those two patches, we would likely be in beta already, or darn close. http://archives.postgresql.org/pgsql-hackers/2010-12/msg01257.php http://archives.postgresql.org/pgsql-hackers/2011-01/msg01209.php http://archives.postgresql.org/pgsql-hackers/2011-01/msg02811.php http://archives.postgresql.org/pgsql-hackers/2011-02/msg00438.php -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
On Fri, Mar 25, 2011 at 8:33 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > The correct question is whether we're ready for beta, and I would say > the answer is clearly no, unless you have a pretty low standard for what > "ready for beta" means. Perhaps it would be suitable to discuss what > the standard for that really ought to be; but I don't agree in the > slightest that we ought to decide based on predetermined calendar dates > rather than the state of the code. Agreed. I think some discussion of which of the things on the open item lists need to be done before beta might be helpful, and we ought to add any items that are not there but are blockers. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
On Sat, Mar 26, 2011 at 12:33 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > > The correct question is whether we're ready for beta, and I would say > the answer is clearly no, unless you have a pretty low standard for what > "ready for beta" means. Perhaps it would be suitable to discuss what > the standard for that really ought to be; but I don't agree in the > slightest that we ought to decide based on predetermined calendar dates > rather than the state of the code. OK >> If we had a hard date for feature freeze, lets have a hard date for >> Beta of +2 months (next time), and +2.5 months now. (I know +1 month >> was suggested, well that's just unrealistic). Beta is a great time to >> resolve difficult decisions, by opening the floor to wider debate and >> feedback. > > The reason we get wider testing during beta is that people have some > confidence (perhaps misplaced) that the database won't eat their data. > Releasing alpha-grade code and calling it beta isn't going to get us > wider testing ... at least, not more than once. I agree that we seem to have slightly different viewpoints on what Beta means. Your definition of Beta sounds like "safe enough to run production systems on", but not trying to put words in your mouth. I think if we have a clear statement of what Beta actually means it would help us know when we've achieved it. And it will also inform potential Beta testers of what we mean by it. The basic point of this post was this: If we wait for the Open Items list to drop to zero, many people are unable to contribute and that means delay. Also, waiting for the Open Items list to drop to zero puts the schedule in the hands of one or two individuals, which is a bad thing. -- Simon Riggs http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
On Sat, Mar 26, 2011 at 1:48 AM, Robert Haas <robertmhaas@gmail.com> wrote: > On Fri, Mar 25, 2011 at 6:18 PM, Simon Riggs <simon@2ndquadrant.com> wrote: > >> The sooner we declare Beta, the sooner people will test. Then we will >> have user feedback, bugs to fix etc.. Everybody is very clearly >> sitting idle. With a longer bug list we will make faster progress to >> release. We're just wasting time. > > I can't resist observing that if you want beta to happen sooner, it > would be better not to commit major and largely unreviewed patches > three weeks after the end of the last CommitFest. Before you insist > that it was reviewed, the version that was actually committed bore so > little resemblance to the versions that were posted earlier that any > earlier review that was done was basically meaningless in terms of > ensuring that the final product was bug free, and it wasn't and isn't. > I complained *repeatedly* about the need to get both collation > support and sync rep finished and committed sooner, for exactly this > reason. We are now reaping the entirely predictable fruit of having > failed to make that happen. But for those two patches, we would > likely be in beta already, or darn close. > > http://archives.postgresql.org/pgsql-hackers/2010-12/msg01257.php > http://archives.postgresql.org/pgsql-hackers/2011-01/msg01209.php > http://archives.postgresql.org/pgsql-hackers/2011-01/msg02811.php > http://archives.postgresql.org/pgsql-hackers/2011-02/msg00438.php There are two responses to your comments. First, you are presuming that the state of those patches must hold up the whole release process. I don't think it should. You want to see it finished before it goes to Beta. I want to see wider input before we consider it finished. In your way of seeing it, you have great input into the decision of what is finished or not. I prefer to open things up to a wider audience, who don't normally get a say until too late. Yes, you do send a great many very long emails and many of them are complaints. If I had read every single word of the many you've written we would be delayed even further. Debating minor points endlessly does not move the world forwards it just delays it. You should trust more that if your points are valid that they will be brought up by another sometime soon. -- Simon Riggs http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
On Sat, Mar 26, 2011 at 9:22 AM, Simon Riggs <simon@2ndquadrant.com> wrote: > First, you are presuming that the state of those patches must hold up > the whole release process. I don't think it should There's not much point in releasing a beta with behaviour that we know we intend to change. All it will do is elicit bug reports that we have to respond to saying "we know, we were going to change that anyways". I think the goal of a beta is to be able to say "we think this is the final behaviour of the next release but we're open to feedback". Once we release the final release we're pretty stuck with the behaviour unless the problems are severe enough to justify changing it. And before the beta, in the alpha releases then it's clear to users that they're seeing work in progress and is most appropriate for people who are already following -hackers. -- greg
On Mar 26, 2011, at 4:27 AM, Simon Riggs <simon@2ndQuadrant.com> wrote: > The basic point of this post was this: If we wait for the Open Items > list to drop to zero, many people are unable to contribute and that > means delay. Also, waiting for the Open Items list to drop to zero > puts the schedule in the hands of one or two individuals, which is a > bad thing. As far as I can tell, everyone is just as free to make suggestions and review patches right as now as they always are. Infact, I do not particularly enjoy slogging through this list of open items. I would be more than happy to have more help.There are plenty of issues there that require real thought, and work, and I have no particular desire to be the onethat fixes them all. You seem to feel that these issues are quite subjective and that the right behavior is altogether unclear. I disagree. Thereare a few things that may fall into that category, but I think for the most part we're fixing bugs and major usabilityproblems. ...Robert
On 03/25/2011 06:18 PM, Simon Riggs wrote: > Judging by the number of new threads about development for 9.2, I > think its time we declared 9.1 Beta. I just had a conversation with > some Debian developers about how PostgreSQL 9.0 got pulled out of > their release because we delayed by 3 weeks. So we missed our slot to > deliver useful new features to our very best supporters by 2 years. I > really hope that was deliberate. > > ISTR Debian on at least on occasion not including a version of Postgres that had been fully released quite some time before their release. We're always going to be missing someone's timeline, not matter what timeline we adopt, anyway. So colour me unimpressed by this whole line of argument. cheers andrew
Greg Stark <gsstark@mit.edu> writes:
> There's not much point in releasing a beta with behaviour that we know
> we intend to change. All it will do is elicit bug reports that we have
> to respond to saying "we know, we were going to change that anyways".
> I think the goal of a beta is to be able to say "we think this is the
> final behaviour of the next release but we're open to feedback".
Yeah, I think this is a productive way to approach the question.
I would put on a couple of extra conditions, though. Something like
this:
* No open issues that are expected to result in user-visible
behavior changes. (Or at least "significant" changes? But then
we have to argue about what's significant --- for instance, are
the questions in the nearby collations-issues thread significant
enough to be beta blockers?)
* No open issues that are expected to result in a catversion bump.
(With pg_upgrade, this is not as critical as it used to be, but
I still think catalog stability is a good indicator of a release's
maturity)
* No known data-loss-causing bugs (duh)
Comments? Any other quality criteria we should have for beta?
regards, tom lane
On Sat, Mar 26, 2011 at 3:46 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > Greg Stark <gsstark@mit.edu> writes: >> There's not much point in releasing a beta with behaviour that we know >> we intend to change. All it will do is elicit bug reports that we have >> to respond to saying "we know, we were going to change that anyways". > >> I think the goal of a beta is to be able to say "we think this is the >> final behaviour of the next release but we're open to feedback". > > Yeah, I think this is a productive way to approach the question. > I would put on a couple of extra conditions, though. Something like > this: > > * No open issues that are expected to result in user-visible > behavior changes. (Or at least "significant" changes? But then > we have to argue about what's significant --- for instance, are > the questions in the nearby collations-issues thread significant > enough to be beta blockers?) > > * No open issues that are expected to result in a catversion bump. > (With pg_upgrade, this is not as critical as it used to be, but > I still think catalog stability is a good indicator of a release's > maturity) > > * No known data-loss-causing bugs (duh) > > Comments? Any other quality criteria we should have for beta? Last 2 are pretty clear. The first one is debatable because of the word "expected". Who decides that? I want more feedback into the project. That can obviously result in changes that are user visible. Users don't complain about non-user visible things. I'd state it the other way around: No open issues that are expected to result in non-user visible architecture changes. -- Simon Riggs http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
On Sat, Mar 26, 2011 at 11:46 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > Greg Stark <gsstark@mit.edu> writes: >> There's not much point in releasing a beta with behaviour that we know >> we intend to change. All it will do is elicit bug reports that we have >> to respond to saying "we know, we were going to change that anyways". > >> I think the goal of a beta is to be able to say "we think this is the >> final behaviour of the next release but we're open to feedback". > > Yeah, I think this is a productive way to approach the question. > I would put on a couple of extra conditions, though. Something like > this: > > * No open issues that are expected to result in user-visible > behavior changes. (Or at least "significant" changes? But then > we have to argue about what's significant --- for instance, are > the questions in the nearby collations-issues thread significant > enough to be beta blockers?) > > * No open issues that are expected to result in a catversion bump. > (With pg_upgrade, this is not as critical as it used to be, but > I still think catalog stability is a good indicator of a release's > maturity) > > * No known data-loss-causing bugs (duh) > > Comments? Any other quality criteria we should have for beta? I think all of these things are pretty subjective, but I broadly agree with the way you've set it out here. Simon is right that it's sometimes reasonable to ship the code as it is and see what feedback we get. But there's a countervailing effect, too: once we ship the code, then people get used to the way it works, and people don't want to change it, even if it's not what they would have picked initially. Magnus mentioned that he was going to upgrade some machine somewhere to alpha5 once it was out. When my jaw fell off he assured me it wasn't a critical system, but still: some people upgrade very early, and once we declare beta, they're going to expect that we aren't going to change too much. And even if they had no such expectation, we don't WANT to change too much, because then we've got to allow more time for beta-testing of the new stuff. It's much easier and much less work to get it right the first time. All that having been said, I think this whole thing may be a tempest in a teapot. Regardless of what the exact criteria are for beta, I think we're getting reasonably close to meeting them. Shaking out the bugs in the collation stuff has taken longer than you originally predicted, but it seems like we're homing in on it; and sync rep has gone from a long list of open items (most of which were reported by Fujii Masao, whose efforts I at least very much appreciate) to a much shorter one. A couple of the other issues are in fact longstanding bugs rather than new regressions in 9.1, and therefore can't be viewed as beta blockers. I see no reason we can't finish the remaining items in the next couple of weeks, barring a sudden influx of new bug reports - especially if a few more people pitch in and help move things forward. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company