Thread: 8.5 release timetable, again
I posted a note about a week ago which drew far less commentary than I expected, regarding the timetable for the release of 8.5. http://archives.postgresql.org/pgsql-hackers/2009-08/msg01256.php Perhaps this is already being discussed on -core, but if so the conclusions haven't been shared publicly. We started the first CommitFest for 8.5 on July 15, 2009. I believe that there is general consensus on a two-month cycle for CommitFests, which means that the second one will start on September 15, 2009, and the third one on November 15, 2009. The open question is whether there will be a fourth or a fifth CommitFest, and what that will do to the time line for the final release. During the 8.4 development cycle, we started the last CommitFest on November 1, 2008 and released 8 months later on July 1, 2009. I would like to think that we could shave a bit of time off of that this time around, because last time the final CommitFest took approximately 4 months. I found that to be pretty ridiculous and I think that I was not the only one. By the end, there were not too many patches left and most of those had been thoroughly reviewed, so there was no obvious role for non-committers to play. There were also many patches that went for weeks, sometimes over a month, without being updated or bounced. I think we can improve this a lot this time around, by applying the same rules to the last CommitFest that we did to the first one: patches must be updated in a timely fashion. If they can't, it's a sign that either the patch author is too busy to work on the patch, or the revisions needed are too extensive for a single CommitFest. Either way, we move on. I am a lot less clear on what we can do to shorten the time we spent in beta. In some ways, the time we spent in beta for 8.4 seems not to have been long enough. Several critical bugs introduced by the infrastructure-changes-for-recovery patch were not fixed until just before release, and a number of planner and other significant bugs have been found since the release and will be fixed in 8.4.1. On the other hand, from my point of view, there was nothing to do during beta. Most of the open items required first and foremost a decision about the best way to fix them, and those decisions really need to be made by (or at least with the consent of) the committers. If we have a substantial open items list again for 8.5, we need to find a way to formalize the handling of that list so that it can be dealt with efficiently and the work distributed among as many people as are willing and able to help. Writing release notes also seemed to take a lot of time, perhaps partly because we didn't know until the very end whether certain large patches were going to be committed. All of the above having been said, I think it's too much to hope that the beta/release cycle is going to get drastically shorter than it was for 8.4. So I think if we decide on three CommitFests, the timetable will end up going something like this: 2009-11-15 Last CommitFest Begins 2009-12-15 Last CommitFest Ends 2010-01-01 Alpha 2010-03-01 Beta 2010-05-01 Release This schedule assumes that instead of having 8 months between start-of-last-CommitFest and release, we'll have five and a half. Assuming that we can avoid the temptation to let the last CommitFest drag on and on and on, that seems achievable. If we go with four CommitFests, we'll probably end up release 45-60 days later, in mid June or start of July. If we go with five CommitFests, we'll probably be looking at September. Personally, I favor the more aggressive schedule of just three CommitFests, because if something goes wrong and the schedule does slip, we should still be able to get the release out the door next year by the same time we did this year. If things go as planned, we'll have the release out the door in time for PGCon. If things go truly excellently and we manage to get the release out before 1-May, then we have that much more time for 8.6 development. On the other hand, if we plan for four Commitfests, we'll be looking at releasing the same time next year that we did this year *if* everything goes as planned. If anything slips, we'll release even later - and that also gets into the summer time, when more people are away. But, some other decision that has consensus is fine too. The most important thing is to MAKE a decision before the passage of time makes one for us. ...Robert
Robert Haas <robertmhaas@gmail.com> writes: > I posted a note about a week ago which drew far less commentary than I > expected, regarding the timetable for the release of 8.5. > http://archives.postgresql.org/pgsql-hackers/2009-08/msg01256.php > Perhaps this is already being discussed on -core, but if so the > conclusions haven't been shared publicly. Core hasn't discussed 8.5 schedule since the discussions that Peter summarized in the message you cited above. I share your concern that "release in time for PGCon" isn't very realistic if we don't get more aggressive about schedule. On the other hand, we didn't get all that much done in this fest. If we cut back to a three-fest schedule I think we may succeed in releasing 8.5 early, but only because there is nothing interesting in it :-(. Dunno where the right balance is. regards, tom lane
On Sun, Aug 23, 2009 at 1:57 AM, Tom Lane<tgl@sss.pgh.pa.us> wrote: > Robert Haas <robertmhaas@gmail.com> writes: >> I posted a note about a week ago which drew far less commentary than I >> expected, regarding the timetable for the release of 8.5. > >> http://archives.postgresql.org/pgsql-hackers/2009-08/msg01256.php > >> Perhaps this is already being discussed on -core, but if so the >> conclusions haven't been shared publicly. > > Core hasn't discussed 8.5 schedule since the discussions that Peter > summarized in the message you cited above. I share your concern that > "release in time for PGCon" isn't very realistic if we don't get more > aggressive about schedule. On the other hand, we didn't get all that > much done in this fest. If we cut back to a three-fest schedule > I think we may succeed in releasing 8.5 early, but only because there > is nothing interesting in it :-(. Dunno where the right balance is. Here is my thinking on that point. We have several major features underway for the September CommitFest, including at least Hot Standby (Simon Riggs), Streaming Replication (Fujii Masao), and Index-Only Scans (Heikki). At the July CommitFest, none of these patches were in a state where they could be seriously reviewed. Simon Riggs and Fujii Masao have both committed to produce reviewable versions by 9/15, and it looks like Heikki will hit that date too, if not beat it. I am assuming that at least Hot Standby and Streaming Replication will likely require two CommitFests to go from the point where they are seriously reviewable to actual commit. So if they hit the 9/15 date, they should make 8.5 even with just three CommitFests. If they don't hit the 9/15 date, then a 3-CommitFest cycle will probably be too short for them to make it in. But if we schedule a fourth CommitFest in January in the hopes of seeing one of those patches committed, then ISTM we're basically speculating that the patch authors will not hit the 9/15 date but that they will hit an 11/15 date. To me, that's an entirely arbitrary and unfounded speculation. On the one hand, both patch authors have said they will hit 9/15, so why should we second-guess them? On the other hand, neither of those patches seems to have made a great deal of progress in the last 7 months, so it's unclear that tacking a few more months onto the schedule will help anything. Furthermore, even if we're right that four CommitFests is the right number to land one of those patches (rather than 3 or 5 or 6 or 7), we're then talking about committing a major feature in the very last CommitFest for this release. We tried that for 8.4 and it wasn't a stunning success. To some degree, what this boils down to is that you can have time-based releases or feature-based releases, but not both. And the problem with feature-based releases in a community environment is that there is no guarantee that patches will be delivered when promised. None of the people developing these major features work for the community and we do not have the ability to control any of their schedules. So saying that we're going to wait for them to be ready is potentially saying that we're going to wait forever. No one is proposing that, but we are sort of trying to read the tea leaves and try to guess when they might be ready and tailor the schedule to it. To me, it seems to make more sense to just decide we're going to ship the release on a time line that works for the project overall. If we get some new features in, good. If they slip a bit past the deadline, well, at least that should hopefully mean that they get committed right at the beginning of the 8.6 cycle. If they slip WAY past the deadline, bummer, but at least we didn't hold up the release to no benefit. I don't think there's anything wrong with having a release that has many small improvements but no major new features, and I think that is the worst that will happen. ...Robert
On sön, 2009-08-23 at 09:37 -0400, Robert Haas wrote: > To some degree, what this boils down to is that you can have > time-based releases or feature-based releases, but not both. Sure. But some people are trying to introduce another subvariant: The conference-circuit-based releases. ;-) It sounds attractive, but it shouldn't trump all other concerns. Consider this instead: Nothing to do during beta? Write your conference slides! ;-) I suggest going with four commit fests. Three is too short. We already started the first one early, which didn't give those involved in the release any time to prepare some patches for it. So with three fests you'd only give the major developers 8 weeks to code something for a yearly release.
On Mon, Aug 24, 2009 at 7:39 AM, Peter Eisentraut<peter_e@gmx.net> wrote: > On sön, 2009-08-23 at 09:37 -0400, Robert Haas wrote: >> To some degree, what this boils down to is that you can have >> time-based releases or feature-based releases, but not both. > > Sure. But some people are trying to introduce another subvariant: The > conference-circuit-based releases. ;-) It sounds attractive, but it > shouldn't trump all other concerns. Consider this instead: Nothing to > do during beta? Write your conference slides! ;-) Oh, gee, what am I speaking on? :-) The main thing that I (can't speak for anyone else) like about getting a release out in time for PGCon is that it happens at about the same time every year, and I kind of like the idea of shooting for a yearly cycle. I'd be just as happy to release every year on Christmas day, but it's harder to get people to bundle a release around then. > I suggest going with four commit fests. Three is too short. We already > started the first one early, which didn't give those involved in the > release any time to prepare some patches for it. So with three fests > you'd only give the major developers 8 weeks to code something for a > yearly release. Well, that's a good point, but 12 weeks out of a 14-month release cycle is only marginally better than 8 weeks out of a 12-month release cycle. If we really want to give people more time to write patches, we need to figure out how to speed up the time from the end of the last CommitFest until release. One thing that I had thought of proposing is that we branch the tree when we go to 8.5-beta and hold the first 8.6 CommitFest at that time - or, failing that, that we hold a DontCommitFest at that time, where we review all of the patches just as we would for a regular CommitFest, but without the committing part. At least for 8.4, it didn't seem like much was happening during beta, at least not much that was discussed on -hackers or could be distributed across the community. I actually don't think that's the ideal solution, though. It's just papering around the problem that the release takes too long and the burden falls on too few people. But we haven't yet come up with any good idea to address that problem. At any rate, I'm OK with 4 CommitFests if that is the consensus, but so far we only have 3 people weighing in on this thread, which is not a consensus for anything. ...Robert
peter_e@gmx.net (Peter Eisentraut) writes: > I suggest going with four commit fests. Three is too short. We already > started the first one early, which didn't give those involved in the > release any time to prepare some patches for it. So with three fests > you'd only give the major developers 8 weeks to code something for a > yearly release. Partial counter-argument... A large portion of the patches in CommitFest #1 represented items that had been deferred from 8.4. So... a) Many of these patches came in with ~6 months of preparation time b) People were always free to start work earlier than CommitFest #1 c) If something requires a *lot* of work, then it may be that it gets deferred so that it comes in as part of CommitFest#1 for 8.6, with the very same characteristics as in a)... I do agree that trying to force coordination with a specific conference in Ottawa seems like a very peculiar sort of forced scheduling. -- select 'cbbrowne' || '@' || 'ca.afilias.info'; Christopher Browne "Bother," said Pooh, "Eeyore, ready two photon torpedoes and lock phasers on the Heffalump, Piglet, meet me in transporter room three"
Christopher Browne <cbbrowne@ca.afilias.info> writes:
> I do agree that trying to force coordination with a specific conference
> in Ottawa seems like a very peculiar sort of forced scheduling.
Well, PGCon is just a convenient concrete target. The real point here
is that we're trying to get the release cycle to end with a release in
the spring. We've grown tired of releasing in the fall and invariably
finding the date slipping into holiday season, so we'd like to try
releasing near the other equinox. (Neither solstice is good, since
you are up against either holidays or vacations taking away the time
of the people who need to be doing the work.)
regards, tom lane
On Mon, Aug 24, 2009 at 5:55 PM, Tom Lane<tgl@sss.pgh.pa.us> wrote: > Christopher Browne <cbbrowne@ca.afilias.info> writes: >> I do agree that trying to force coordination with a specific conference >> in Ottawa seems like a very peculiar sort of forced scheduling. > > Well, PGCon is just a convenient concrete target. The real point here > is that we're trying to get the release cycle to end with a release in > the spring. Yeah, conference-based releases is just a proxy for time-based releases. It's nice to have something to be happy about at the conference too. And it's a convenient time to start talking about the next release when you're all face-to-face. -- greg http://mit.edu/~gsstark/resume.pdf
All, > Yeah, conference-based releases is just a proxy for time-based > releases. It's nice to have something to be happy about at the > conference too. And it's a convenient time to start talking about the > next release when you're all face-to-face. On the one hand: I'd say that we go for the 3-CF release. I think we need to prove that we can do a time-based release once before a lot of people on this list will believe in it. If we do 4 CFs, we're in danger of still being in beta in late May ... and once conference and vacation season start, things get a lot slower.Mind you, it's possible that we can shorten the finalCF and beta this time, but I wouldn't want to count on it. If we *can* shorten them, then 8.6 can have 4CFs. But we won't know until after we've done it. On the other hand: I think if we do another release without Standby/replication, we'll start to lose a lot of users. People are waiting on that, and a lot of folks were expecting it in 8.4. Therefore: I think, 3CFs, but we go all-out to get Standby/Replication into 8.5 in the next month. So, every committer/major hacker on this list should pitch in to get those features done. So, is there someone here who could be helping with HS/SR and isn't? Why not? -- Josh Berkus PostgreSQL Experts Inc. www.pgexperts.com
On Mon, Aug 24, 2009 at 1:48 PM, Josh Berkus<josh@agliodbs.com> wrote: > Therefore: I think, 3CFs, but we go all-out to get Standby/Replication > into 8.5 in the next month. So, every committer/major hacker on this > list should pitch in to get those features done. > > So, is there someone here who could be helping with HS/SR and isn't? > Why not? Unfortunately, neither Simon nor Fujii Masao is publishing a git repository of their work or regular updates to their patch set. Therefore, no one else can contribute code, and in fact no one else can even provide a review, because there is nothing to look at. I don't think that there would be any problem getting these patches reviewed/committed outside of the regular CommitFest cycle, but there's simply no place to start. I have volunteered to work on Hot Standby and have merged it up to CVS HEAD several times, and cleaned out a bit of other cruft, but the scope of the task appears to exceed the time I have available. It seems to me that other people could clone my git repo (or that branch) and submit patches against it (or ask me to pull from a branch that they publish), but will Simon incorporate those patches (assuming that they don't suck) into his next version, or ignore them? It's not really clear. As I've said before, I am presently of the opinion that Streaming Replication has little chance of making it into 8.5. This opinion is vulnerable to contrary evidence, like a new version of the patch showing up that shows massive progress. But the patch was bounced from CF 2009-07 for a whole series of architectural problems which have to be addressed before we can even get to implementation details, bugs, documentation, etc. Hot Standby is in better shape but amount of code cleanup needed is substantial and there is also quite a bit of 'git diff master | grep XXX' that needs to be gone through. ...Robert
Josh Berkus <josh@agliodbs.com> writes:
> So, is there someone here who could be helping with HS/SR and isn't?
> Why not?
You mean, other than Simon's hands-off attitude?
regards, tom lane
Josh Berkus wrote: > All, > > > Yeah, conference-based releases is just a proxy for time-based > > releases. It's nice to have something to be happy about at the > > conference too. And it's a convenient time to start talking about the > > next release when you're all face-to-face. > > On the one hand: > > I'd say that we go for the 3-CF release. I think we need to prove that > we can do a time-based release once before a lot of people on this list > will believe in it. > > If we do 4 CFs, we're in danger of still being in beta in late May ... > and once conference and vacation season start, things get a lot slower. > Mind you, it's possible that we can shorten the final CF and beta this > time, but I wouldn't want to count on it. > > If we *can* shorten them, then 8.6 can have 4CFs. But we won't know > until after we've done it. > > On the other hand: > > I think if we do another release without Standby/replication, we'll > start to lose a lot of users. People are waiting on that, and a lot of > folks were expecting it in 8.4. That is a slightly alarmist. Who are we going to lose these users to? -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. +
Robert Haas wrote: > As I've said before, I am presently of the opinion that Streaming > Replication has little chance of making it into 8.5. This opinion is > vulnerable to contrary evidence, like a new version of the patch > showing up that shows massive progress. But the patch was bounced > from CF 2009-07 for a whole series of architectural problems which > have to be addressed before we can even get to implementation details, > bugs, documentation, etc. Hot Standby is in better shape but amount > of code cleanup needed is substantial and there is also quite a bit of > 'git diff master | grep XXX' that needs to be gone through. I agree. I think it is unlikely we will have anything ready to commit for Streaming Replication or Hot Standby for the next commit-fest in mid-September, and if we go for a 3-CF (commit fest) release, that gives us only one final CF to get those features accepted, again unlikely. We are either going to need to go to a 4-CF release, change the way we are developing these patches, or both to get either in 8.5. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. +
On Mon, Aug 24, 2009 at 4:56 PM, Bruce Momjian<bruce@momjian.us> wrote: > Robert Haas wrote: >> As I've said before, I am presently of the opinion that Streaming >> Replication has little chance of making it into 8.5. This opinion is >> vulnerable to contrary evidence, like a new version of the patch >> showing up that shows massive progress. But the patch was bounced >> from CF 2009-07 for a whole series of architectural problems which >> have to be addressed before we can even get to implementation details, >> bugs, documentation, etc. Hot Standby is in better shape but amount >> of code cleanup needed is substantial and there is also quite a bit of >> 'git diff master | grep XXX' that needs to be gone through. > > I agree. I think it is unlikely we will have anything ready to commit > for Streaming Replication or Hot Standby for the next commit-fest in > mid-September, and if we go for a 3-CF (commit fest) release, that gives > us only one final CF to get those features accepted, again unlikely. We > are either going to need to go to a 4-CF release, change the way we are > developing these patches, or both to get either in 8.5. I don't think a 4-CF release is going to help. It's just going to be 2 more months before everything else that has been done gets released.Call me a pessimist if you will, but zero times anarbitrary number of CommitFests is still zero. The only solution here is to get more people working on these patches.I have volunteered to work on HS and would also bewilling to work on SR. Work can be reviewing or actual code. But I cannot work on a patch I cannot see, and neither can anyone else. ...Robert
Robert Haas wrote: > On Mon, Aug 24, 2009 at 4:56 PM, Bruce Momjian<bruce@momjian.us> wrote: > > Robert Haas wrote: > >> As I've said before, I am presently of the opinion that Streaming > >> Replication has little chance of making it into 8.5. ?This opinion is > >> vulnerable to contrary evidence, like a new version of the patch > >> showing up that shows massive progress. ?But the patch was bounced > >> from CF 2009-07 for a whole series of architectural problems which > >> have to be addressed before we can even get to implementation details, > >> bugs, documentation, etc. ?Hot Standby is in better shape but amount > >> of code cleanup needed is substantial and there is also quite a bit of > >> 'git diff master | grep XXX' that needs to be gone through. > > > > I agree. ?I think it is unlikely we will have anything ready to commit > > for Streaming Replication or Hot Standby for the next commit-fest in > > mid-September, and if we go for a 3-CF (commit fest) release, that gives > > us only one final CF to get those features accepted, again unlikely. ?We > > are either going to need to go to a 4-CF release, change the way we are > > developing these patches, or both to get either in 8.5. > > I don't think a 4-CF release is going to help. It's just going to be > 2 more months before everything else that has been done gets released. > Call me a pessimist if you will, but zero times an arbitrary number > of CommitFests is still zero. > > The only solution here is to get more people working on these patches. > I have volunteered to work on HS and would also be willing to work on > SR. Work can be reviewing or actual code. But I cannot work on a > patch I cannot see, and neither can anyone else. Agreed, so we fall back to "change the way we are developing these patches". I am hesistant to jump into managing these patches until they get their shot by their original authors in the September CF, but managing them starting in mid-October will probably be too late for them to get into 8.5. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. +
On Monday 24 August 2009 3:51:31 pm Bruce Momjian wrote: > > folks were expecting it in 8.4. > > That is a slightly alarmist. Who are we going to lose these users to? the insane asylum?
On Mon, Aug 24, 2009 at 04:51:31PM -0400, Bruce Momjian wrote: > Josh Berkus wrote: > > All, > > > > > Yeah, conference-based releases is just a proxy for time-based > > > releases. It's nice to have something to be happy about at the > > > conference too. And it's a convenient time to start talking > > > about the next release when you're all face-to-face. > > > > On the one hand: > > > > I'd say that we go for the 3-CF release. I think we need to prove > > that we can do a time-based release once before a lot of people on > > this list will believe in it. > > > > If we do 4 CFs, we're in danger of still being in beta in late May > > ... and once conference and vacation season start, things get a > > lot slower. Mind you, it's possible that we can shorten the final > > CF and beta this time, but I wouldn't want to count on it. > > > > If we *can* shorten them, then 8.6 can have 4CFs. But we won't > > know until after we've done it. > > > > On the other hand: > > > > I think if we do another release without Standby/replication, > > we'll start to lose a lot of users. People are waiting on that, > > and a lot of folks were expecting it in 8.4. > > That is a slightly alarmist. Who are we going to lose these users > to? Sadly, to one of the MySQL forks. This is one of those cases (cf. the current thread on -advocacy) where pointy-hair-friendliness can really help or hurt us. Cheers, David. -- David Fetter <david@fetter.org> http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fetter@gmail.com Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate
>> I think if we do another release without Standby/replication, we'll >> start to lose a lot of users. People are waiting on that, and a lot of >> folks were expecting it in 8.4. > > That is a slightly alarmist. Who are we going to lose these users to? Drizzle. MySQL forks. CouchDB. Any database which has replication which you don't need a professional DBA to understand. Whether or not it works. -- Josh Berkus PostgreSQL Experts Inc. www.pgexperts.com
Josh Berkus <josh@agliodbs.com> writes:
>> That is a slightly alarmist. Who are we going to lose these users to?
> Drizzle. MySQL forks. CouchDB. Any database which has replication
> which you don't need a professional DBA to understand. Whether or not
> it works.
You haven't explained why we'd lose such folk next year when we haven't
lost them already. MySQL has had replication (or at least has checked
off the bullet point ;-)) for years. I'd be seriously surprised if any
of the forks will offer significantly better replication than is there
now, so the competitive situation is not changing in that regard.
It is true that we're missing a chance to pull some folks away while
the situation on that side of the fence is so messy. But I don't see
our situation getting worse because of that, just not getting better.
regards, tom lane
On Mon, Aug 24, 2009 at 08:02:31PM -0400, Tom Lane wrote: > Josh Berkus <josh@agliodbs.com> writes: > >> That is a slightly alarmist. Who are we going to lose these > >> users to? > > > Drizzle. MySQL forks. CouchDB. Any database which has > > replication which you don't need a professional DBA to understand. > > Whether or not it works. > > You haven't explained why we'd lose such folk next year when we > haven't lost them already. MySQL has had replication (or at least > has checked off the bullet point ;-)) for years. I'd be seriously > surprised if any of the forks will offer significantly better > replication than is there now, so the competitive situation is not > changing in that regard. > > It is true that we're missing a chance to pull some folks away while > the situation on that side of the fence is so messy. But I don't > see our situation getting worse because of that, just not getting > better. "Not getting better," isn't a situation to be dismissed lightly. In FLOSS, as I've seen it, a project whose adoption isn't growing is dying. Cheers, David. -- David Fetter <david@fetter.org> http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fetter@gmail.com Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate
On Mon, Aug 24, 2009 at 10:15 PM, David Fetter<david@fetter.org> wrote: > On Mon, Aug 24, 2009 at 08:02:31PM -0400, Tom Lane wrote: >> Josh Berkus <josh@agliodbs.com> writes: >> >> That is a slightly alarmist. Who are we going to lose these >> >> users to? >> >> > Drizzle. MySQL forks. CouchDB. Any database which has >> > replication which you don't need a professional DBA to understand. >> > Whether or not it works. >> >> You haven't explained why we'd lose such folk next year when we >> haven't lost them already. MySQL has had replication (or at least >> has checked off the bullet point ;-)) for years. I'd be seriously >> surprised if any of the forks will offer significantly better >> replication than is there now, so the competitive situation is not >> changing in that regard. >> >> It is true that we're missing a chance to pull some folks away while >> the situation on that side of the fence is so messy. But I don't >> see our situation getting worse because of that, just not getting >> better. > > "Not getting better," isn't a situation to be dismissed lightly. In > FLOSS, as I've seen it, a project whose adoption isn't growing is > dying. You may be right, but I'm not sure that it has much to do with the ostensible topic of this thread. The reason why these features have not already been committed is because they are not done. The question of why they are not done and/or why more people aren't working on them has been asked and answered. Figuring out some concrete steps that we can take to address those issues is probably a better use of time than trying to define whether these features are really important or really, really important. ...Robert
On Aug 24, 2009, at 9:46 PM, Robert Haas wrote: > On Mon, Aug 24, 2009 at 10:15 PM, David Fetter<david@fetter.org> > wrote: >> On Mon, Aug 24, 2009 at 08:02:31PM -0400, Tom Lane wrote: >>> Josh Berkus <josh@agliodbs.com> writes: >>>>> That is a slightly alarmist. Who are we going to lose these >>>>> users to? >>> >>>> Drizzle. MySQL forks. CouchDB. Any database which has >>>> replication which you don't need a professional DBA to understand. >>>> Whether or not it works. >>> >>> You haven't explained why we'd lose such folk next year when we >>> haven't lost them already. MySQL has had replication (or at least >>> has checked off the bullet point ;-)) for years. I'd be seriously >>> surprised if any of the forks will offer significantly better >>> replication than is there now, so the competitive situation is not >>> changing in that regard. >>> >>> It is true that we're missing a chance to pull some folks away while >>> the situation on that side of the fence is so messy. But I don't >>> see our situation getting worse because of that, just not getting >>> better. One possible reason that replication is more critical now than it was a year ago is the rise in cloud computing. The ability to fire up instances on demand is much more useful when some of those boxes can be database servers and those databases servers can replicate the primary database and start doing something useful. As far as I can tell this one feature alone is the one thing that makes it hard to convince people to migrate away from Mysql despite it's demonstrable inferiority in many other areas. Postgres seems to be winning mindshare as the "real" and reliable database of choice for people who are serious about their data. But for many, many businesses (many of whom are really not that serious about their data) easy to set up replication is just too big of a draw, such that you can't get them to consider anything without it. I don't know if current postgres users are really going to switch over existing projects that were built on postgres, but for new apps running on EC2 or similar I would not be surprised to see people choosing mysql over postgres solely on this one issue. Databases scalability is becoming and issue for more and more businesses and others are filling in the gap. If postgres could combine it's current deserved reputation for having a robust feature set, standards compliance, high performance, reliability, stability, etc, with easy to use replication it would be be a slam dunk, no-brainer decision to go with postgres on just about anything. Just my 2 cents. Rick P.S. I don't actually use mysql anywhere but I know many who do and replication is always the sticking point.
Le mercredi 26 août 2009 à 01:36 -0600, Rick Gigger a écrit : > One possible reason that replication is more critical now than it > was > a year ago is the rise in cloud computing. The ability to fire up > instances on demand is much more useful when some of those boxes can > be database servers and those databases servers can replicate the > primary database and start doing something useful. As far as I can > tell this one feature alone is the one thing that makes it hard to > convince people to migrate away from Mysql despite it's demonstrable > inferiority in many other areas. I think you should have a deep look at these two manuals that I wrote for Drupal: Guidelines for writing MySQL and PostgreSQL compliant SQL http://drupal.org/node/555514 and Quidelines for writing SQL efficient code: http://drupal.org/node/559302 I have been using PostgreSQL since 1998. I took part in the development of pgAdmin 1 and pgAdmin 2. I am very proud of the PostgreSQL community, but I think it should fix some important issues in the domain of SQL compliance and compatibility. When reading developers comments on Drupal web site, it seems that there is deep misunderstanding between developers and SQL databases. I would say that 1% of developers know database technology. For example, most Drupal developers think that an INNER JOIN is faster than a LEFT JOIN. The reality of facts is that developers will not even test PostgreSQL and stop using it after the first SQL warning or error. So I would recommend focussing on usability. Then you can work on replication and materilized views. You probably know that a cloud requires several computers. With materialized view, a single computer can perform 100 times better. I see plenty of of possibilities to improve speed using materialized views. But first and firstly ... focus on usability. Then make a pre-release of a PostgreSQL 8.4.2 release or so ... We need to wipe out this MySQL issue once for all. If there is a compat MySQL mode or functions, then include it in core. This is too important for PostgreSQL success. Why MySQL usability is achieved add materialized views and MySQL is dead in the performance statistics, beaten 99% by PostgreSQL. Kind regards, Jean-Michel
Jean-Michel Pouré<jm@poure.com> wrote: > focus on usability. It's not clear to me what you feel is needed. That could mean many things.... -Kevin
Le mercredi 26 août 2009 à 09:30 -0500, Kevin Grittner a écrit : > It's not clear to me what you feel is needed. That could mean many > things.... Dear Kevin, I rarely post on Hackers, so I will try to explain: * I use PostgreSQL since 1998. * I took part in the development of pgAdmin 1&2. * I love PostgreSQL and I believe MySQL sucks. Recently I was forced to use MySQL for Kdenlive.org project and the database sometimes stops responding sending nothing or crap. I believe that if you use MySQL in your company for a paid work, you can die of a heart attack. But at the same time: * I rewrote very long and tedious queries from PHPBB. Maybe 100 of them. They are now part of PhpBB3. I drove all queries below 30 ms and this enables PhpBB to scale easily. I consider this is my work. * I think Drupal queries presently have performance problems. If I wanted, I would be able to drive down Drupal web site, using a collection of simple queries on projects, forum and comments. But I don't of course. This is why I wrote http://drupal.org/node/559302 and http://drupal.org/node/555514 Everytime I try a new Drupal module under PostgreSQL, I run into tiny SQL problems ranging from error to performance drop. The most problematic problem is http://drupal.org/node/559986 To fix a problem, I need to open a thread on Drupal web site and wait for the maintainer to discuss and commit. To give an example, Drupal includes a query optimizer written in PHP, which sometimes adds "DISTINCT" to queries. In the forum, some incredible query fetches 22000 rows, copies the rows to an arrays and then computes the array. This allows to display previous and next message. But we are not going to change the world of MySQL users, which believe they know SQL programming and in reality are complete beginners, who like to boast about farms and replication, just like Windows users like to collect Adobe products on DVDs and discuss with friends about them. IMHO for what I know from the porting work (I worked heavily on PHPBB3 and now Drupal), the first goal is to achieve compatibility with issues mentioned there: http://drupal.org/node/555514 and add mysql compat functions in PostgreSQL core without breaking existing code. Then I can insure that 99% of MySQL compatibility problems will be behind. When this is achieve, we will be able to start education of developers. And this will take another decade. To win over MySQL, the best is to work on materialized views. There are very good articles available from hackers. Why not port to C. Materialized which which update when the data is needed would be perfect. Then we can convince Drupal hackers to add views in the schema. The trick would be that MySQL would support normal views, whereas we would also support materialized. We can do the same with nearly all available frameworks: PhpBB, etc ... Web apps are 95% of PostgreSQL possible users. Kind regards, Jean-Michel
Jean-Michel Pouré wrote: > > Everytime I try a new Drupal module under PostgreSQL, I run into tiny > SQL problems ranging from error to performance drop. The most > problematic problem is http://drupal.org/node/559986 > I strongly suspect this post badly mis-diagnoses the problem. > > IMHO for what I know from the porting work (I worked heavily on PHPBB3 > and now Drupal), the first goal is to achieve compatibility with issues > mentioned there: http://drupal.org/node/555514 and add mysql compat > functions in PostgreSQL core without breaking existing code. > That might be your goal, but it's not the community's goal, I believe. There are already external projects for compatibility libraries. You are never going to achieve 100% compatibility. > > To win over MySQL, the best is to work on materialized views. There are > very good articles available from hackers. Why not port to C. > Materialized which which update when the data is needed would be > perfect. > IIRC some work was being done on materialised views. > > Web apps are 95% of PostgreSQL possible users. > Most applications these days have a web front end. But that doesn't mean the database needs to be terribly aware of them. To the database, a web server is just another client. cheers andrew
On Aug 26, 2009, at 11:18 , Jean-Michel Pouré wrote: > Web apps are 95% of PostgreSQL possible users. Where does this figure come from? Michael Glaesemann grzm seespotcode net
Jean-Michel Pouré<jm@poure.com> wrote: > Kevin Grittner a écrit : >> It's not clear to me what you feel is needed. > http://drupal.org/node/559302 These look like performance issues. > http://drupal.org/node/555514 These are MySQL compatibility issues. So when you talk about focusing on usablility improvements you mean that priority should be given to supporting MySQL-specific syntax extensions and ensuring that there are no queries where the MySQL optimizer comes up with a more efficient plan than PostgreSQL? One concern I have is that you don't mention PostgreSQL configuration in your performance advice, and I seem to remember you said that you didn't tune your postgresql.conf file beyond boosting the shared_buffers setting. If that's true, you might be somewhat surprised with the performance improvements if you tweak just a few other settings. You might want to see what suggestions you get from: http://pgfoundry.org/projects/configurator/ -Kevin
Robert Haas wrote: > I am assuming that at least Hot Standby and Streaming Replication will > likely require two CommitFests to go from the point where they are > seriously reviewable to actual commit. So if they hit the 9/15 date, > they should make 8.5 even with just three CommitFests. If they don't > hit the 9/15 date, then a 3-CommitFest cycle will probably be too > short for them to make it in. But if we schedule a fourth CommitFest > in January in the hopes of seeing one of those patches committed, then > ISTM we're basically speculating that the patch authors will not hit > the 9/15 date but that they will hit an 11/15 date. My concern is not just with those features, but with the whole ratio of the window for new work to the total development cycle. That ratio keeps going down and the time the tree is effectively frozen to new features keeps going up. I'd like to see us keep the tree open as long as possible but be much more ruthless about chopping off things that aren't ready at the end. That way we can quickly get to a beta and get on with the next cycle. I realise the idea is that significant features must be submitted by the penultimate CF, but I'm not too sure how well that's going to work in practice. That just seems like we're relabelling things rather than a fundamental change. At the very least my vote goes for four CFs rather than three. cheers andrew
On Wed, Aug 26, 2009 at 12:07 PM, Andrew Dunstan<andrew@dunslane.net> wrote: > Robert Haas wrote: >> >> I am assuming that at least Hot Standby and Streaming Replication will >> likely require two CommitFests to go from the point where they are >> seriously reviewable to actual commit. So if they hit the 9/15 date, >> they should make 8.5 even with just three CommitFests. If they don't >> hit the 9/15 date, then a 3-CommitFest cycle will probably be too >> short for them to make it in. But if we schedule a fourth CommitFest >> in January in the hopes of seeing one of those patches committed, then >> ISTM we're basically speculating that the patch authors will not hit >> the 9/15 date but that they will hit an 11/15 date. > > My concern is not just with those features, but with the whole ratio of the > window for new work to the total development cycle. That ratio keeps going > down and the time the tree is effectively frozen to new features keeps going > up. I think that's a very valid concern. Against that, if release cycles become very long, then features can hit the tree more of the time, but they don't get into a released version for ages. > I'd like to see us keep the tree open as long as possible but be much > more ruthless about chopping off things that aren't ready at the end. That > way we can quickly get to a beta and get on with the next cycle I'm happy to assist with that, but recall that even after we ended CF 2008-11 another four months went by before release. That's a whole lotta time for the tree to be closed right there. > I realise > the idea is that significant features must be submitted by the penultimate > CF, but I'm not too sure how well that's going to work in practice. That > just seems like we're relabelling things rather than a fundamental change. > At the very least my vote goes for four CFs rather than three. Fair enough, more votes are good. ...Robert
Andrew Dunstan <andrew@dunslane.net> wrote: > the window for new work to the total development cycle. That ratio > keeps going down and the time the tree is effectively frozen to new > features keeps going up. I'd like to see us keep the tree open as > long as possible but be much more ruthless about chopping off things > that aren't ready at the end. That way we can quickly get to a beta > and get on with the next cycle. I realise the idea is that > significant features must be submitted by the penultimate CF, but > I'm not too sure how well that's going to work in practice. That > just seems like we're relabelling things rather than a fundamental > change. At the very least my vote goes for four CFs rather than > three. Unless the community can reduce the time between the start of the last commit-fest and the release, you're limited to an average of four months of programming time per year for new features (assuming that people are observing the rules about what they should be doing during commit-fests and beta testing). If you want to move the next release back into Spring rather than Summer (which is the season in which 8.4 was released -- at least of those of us in the Northern Hemisphere), you would need to shorten that to three months for this release. Unless... Both the ruthless cutting of anything not totally ready at the end of a commit-fest, *and* reducing the time from the end of the last commit-fest to release would be needed to get that up to five months per year. We obviously don't want less testing during the beta cycle, but delaying the release while the release notes are developed at the end of the cycle seems like an obvious target for improvement. I'd bet there are others, though I don't know what they are.... -Kevin
> Robert Haas wrote:
>> I am assuming that at least Hot Standby and Streaming Replication will
>> likely require two CommitFests to go from the point where they are
>> seriously reviewable to actual commit.
FWIW, I think that both HS and SR are special cases: if we ever see
reviewable patches for them, people will probably be willing to work
on them outside the CommitFest framework. We shouldn't be setting
the schedule with the idea that those will only be dealt with in CFs.
To my mind the CF process is for dealing with "run of the mill" patches.
Andrew Dunstan <andrew@dunslane.net> writes:
> My concern is not just with those features, but with the whole ratio of
> the window for new work to the total development cycle. That ratio keeps
> going down and the time the tree is effectively frozen to new features
> keeps going up.
Yup. This is a huge problem and we need to deal with it somehow. At
the same time, I'm worried that our beta testing process is already
inadequate. We've found several rather embarrassing bugs in 8.4, for
instance, things that should have been found in beta IMO. Shortening
beta or encouraging people to start next-cycle development instead of
testing doesn't seem like a wise move. You can't just develop all
the time, you have to test & debug too ...
regards, tom lane
Tom Lane wrote: > Andrew Dunstan <andrew@dunslane.net> writes: > >> My concern is not just with those features, but with the whole ratio of >> the window for new work to the total development cycle. That ratio keeps >> going down and the time the tree is effectively frozen to new features >> keeps going up. >> > > Yup. This is a huge problem and we need to deal with it somehow. At > the same time, I'm worried that our beta testing process is already > inadequate. We've found several rather embarrassing bugs in 8.4, for > instance, things that should have been found in beta IMO. Shortening > beta or encouraging people to start next-cycle development instead of > testing doesn't seem like a wise move. You can't just develop all > the time, you have to test & debug too ... > > > Perhaps some more formalised beta program would be useful. I have at least one client who could probably be persuaded to devote some resources to Beta testing. Maybe we need a Beta Program co-ordinator or some such animal. I suspect that plenty of possible beta testers don't even know when we go beta. cheers andrew
Andrew Dunstan escribió: > Perhaps some more formalised beta program would be useful. I have at > least one client who could probably be persuaded to devote some > resources to Beta testing. Maybe we need a Beta Program co-ordinator > or some such animal. I suspect that plenty of possible beta testers > don't even know when we go beta. This seems a good idea. Possibly pushing the betas more aggresively to current users would make them tested not only by PG hackers ... -- Alvaro Herrera http://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc.
Andrew Dunstan <andrew@dunslane.net> wrote: > Perhaps some more formalised beta program would be useful. I have > at least one client who could probably be persuaded to devote some > resources to Beta testing. Maybe we need a Beta Program co-ordinator > or some such animal. I suspect that plenty of possible beta testers > don't even know when we go beta. In some shops it would be necessary to have a date set months in advance of the start of the beta to have any chance of getting managers to assign hardware and staff for a really good test. I usually have to work in tests on my own time on whatever hardware happens to be in standby status, when I can get it. Hitting such a date would seem to require a significant change from prior releases; although the just-completed commit-fest could be taken as evidence that such a thing is possible. -Kevin
On Wed, Aug 26, 2009 at 1:01 PM, Tom Lane<tgl@sss.pgh.pa.us> wrote: > Yup. This is a huge problem and we need to deal with it somehow. At > the same time, I'm worried that our beta testing process is already > inadequate. We've found several rather embarrassing bugs in 8.4, for > instance, things that should have been found in beta IMO. Shortening > beta or encouraging people to start next-cycle development instead of > testing doesn't seem like a wise move. You can't just develop all > the time, you have to test & debug too ... Sure, but an aimless mandate to do testing for 4 (or 8, or 12) months doesn't necessarily buy you much, either. I'm good at focused activity - but there was nothing focused about 8.4 beta that I could see. Maybe we need some kind of TestFest process. ...Robert
Dear Kevin > So when you talk about focusing on usablility improvements you mean > that priority should be given to supporting MySQL-specific syntax > extensions and ensuring that there are no queries where the MySQL > optimizer comes up with a more efficient plan than PostgreSQL? Yes. PostgreSQL should be able to run MySQL code quoted here: This is a prerequisite for people to be willing to test and adopt PostgreSQL. People are not willing to debug frameworks like Drupal and port them to PostgreSQL. We are quite alone and lost. > One concern I have is that you don't mention PostgreSQL configuration > in your performance advice, and I seem to remember you said that you > didn't tune your postgresql.conf file beyond boosting the > shared_buffers setting. If that's true, you might be somewhat > surprised with the performance improvements if you tweak just a few > other settings. shared_buffer 24M. Kind regards, Jean-Michel
Alvaro Herrera wrote: > This seems a good idea. Possibly pushing the betas more aggresively to > current users would make them tested not only by PG hackers ... Isn't this the purpose of the new alpha releases, at lease to some extent.
"Matthew T. O'Connor" <matthew@zeut.net> writes:
> Alvaro Herrera wrote:
>> This seems a good idea. Possibly pushing the betas more aggresively to
>> current users would make them tested not only by PG hackers ...
> Isn't this the purpose of the new alpha releases, at lease to some extent.
The alpha releases as currently constituted are practically the exact
opposite of what's being suggested here :-(. We are pushing them out
without very much advertisement, and certainly without any formal
program to encourage significant testing. That's more or less forced by
the decision that alpha releases should be low-overhead, but I think
we're unlikely to get wide-ranging test coverage that way.
regards, tom lane
2009/8/26 Jean-Michel Pouré <jm@poure.com>: > Dear Kevin > >> So when you talk about focusing on usablility improvements you mean >> that priority should be given to supporting MySQL-specific syntax >> extensions and ensuring that there are no queries where the MySQL >> optimizer comes up with a more efficient plan than PostgreSQL? > > Yes. PostgreSQL should be able to run MySQL code quoted here: > > This is a prerequisite for people to be willing to test and adopt > PostgreSQL. People are not willing to debug frameworks like Drupal and > port them to PostgreSQL. We are quite alone and lost. Er, so we should debug Drupal for them? I find it difficult to believe that's the best use of our time. >> One concern I have is that you don't mention PostgreSQL configuration >> in your performance advice, and I seem to remember you said that you >> didn't tune your postgresql.conf file beyond boosting the >> shared_buffers setting. If that's true, you might be somewhat >> surprised with the performance improvements if you tweak just a few >> other settings. > > shared_buffer 24M. That doesn't actually respond to anything he said in that paragraph. ...Robert
>> So when you talk about focusing on usablility improvements you mean >> that priority should be given to supporting MySQL-specific syntax >> extensions and ensuring that there are no queries where the MySQL >> optimizer comes up with a more efficient plan than PostgreSQL? Well, I'd be interested in this just because I *always* want to improve the optimizer. I think these cases demonstrate places we could improve our optimizer still further, just as looking for the same cases with Oracle or SQL Server does. > Yes. PostgreSQL should be able to run MySQL code quoted here: > > This is a prerequisite for people to be willing to test and adopt > PostgreSQL. People are not willing to debug frameworks like Drupal and > port them to PostgreSQL. We are quite alone and lost. First, off, the evidence is against that; people are doing the work to port things. And creating new projects which are based on Postgres. Second, we're not going to support MySQL's *bugs* and *bad design decisions* which is what lazy developers actually want; they want something exactly the same as MySQL, including bugs. If they want that, they can use MySQL. We are not MySQL, and trying to out-MySQL MySQL is stupid, just as it would be to copy Oracle exactly. Now, there are things which MySQL does better which we should fix, because they are real problems for our users who already like PostgreSQL. These include simple replication, upgrade in place, driver maintenance, covering indexes, MERGE, etc. But we'll do these because they make *Postgres* better, not because MySQL has them. -- Josh Berkus PostgreSQL Experts Inc. www.pgexperts.com
On Wed, Aug 26, 2009 at 02:46:43PM -0400, Tom Lane wrote: > "Matthew T. O'Connor" <matthew@zeut.net> writes: > > Alvaro Herrera wrote: > >> This seems a good idea. Possibly pushing the betas more > >> aggresively to current users would make them tested not only by > >> PG hackers ... > > > Isn't this the purpose of the new alpha releases, at lease to some > > extent. > > The alpha releases as currently constituted are practically the > exact opposite of what's being suggested here :-(. We are pushing > them out without very much advertisement, and certainly without any > formal program to encourage significant testing. That's more or > less forced by the decision that alpha releases should be > low-overhead, but I think we're unlikely to get wide-ranging test > coverage that way. How would you recommend that this change? Cheers, David. -- David Fetter <david@fetter.org> http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fetter@gmail.com Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate
David Fetter <david@fetter.org> writes:
> On Wed, Aug 26, 2009 at 02:46:43PM -0400, Tom Lane wrote:
>> The alpha releases as currently constituted are practically the
>> exact opposite of what's being suggested here :-(. We are pushing
>> them out without very much advertisement, and certainly without any
>> formal program to encourage significant testing. That's more or
>> less forced by the decision that alpha releases should be
>> low-overhead, but I think we're unlikely to get wide-ranging test
>> coverage that way.
> How would you recommend that this change?
As far as the alpha releases go, I wouldn't --- I see no evidence that
we have the manpower to formalize them any more than they are now.
I do like the idea of trying to reach out to more beta testers and
manage that phase more aggressively. Maybe if we can make something
happen there, it would be possible to do the same for alphas later.
regards, tom lane
On Wednesday 26 August 2009 12:17:25 pm Josh Berkus wrote: > > Yes. PostgreSQL should be able to run MySQL code quoted here: > > > > This is a prerequisite for people to be willing to test and adopt > > PostgreSQL. People are not willing to debug frameworks like Drupal and > > port them to PostgreSQL. We are quite alone and lost. > > First, off, the evidence is against that; people are doing the work to > port things. And creating new projects which are based on Postgres. Yeah the above is pretty much a non starter anymore. We out rank MySQL on several polls in terms of popularity. Our channel is growing whereas the MySQL one is shrinking. Our community isn't splintered and even projects like Drupal are making it a specific point to fix the original mysql centric design. > Now, there are things which MySQL does better which we should fix, > because they are real problems for our users who already like > PostgreSQL. These include simple replication, upgrade in place, driver > maintenance, covering indexes, MERGE, etc. But we'll do these because > they make *Postgres* better, not because MySQL has them. Agreed. Joshua D. Drake -- PostgreSQL.org Major Contributor Command Prompt, Inc : 503-667-4564 - http://www.commandprompt.com/ Since 1997, Consulting, Development, Support, Training
> Second, we're not going to support MySQL's *bugs* and *bad design > decisions* which is what lazy developers actually want; they want > something exactly the same as MySQL, including bugs. If they want > that, > they can use MySQL. We are not MySQL, and trying to out-MySQL MySQL > is > stupid, just as it would be to copy Oracle exactly. I understand what you mean. To tell you how lazy MySQL people are is my last experience in the Drupal world. In short, on my devel server, Drupal previous/next link display SQL script returns 21.000 rows. Of course, it should return only two rows. The 21.000 rows are then processed step by step by a PHP script. I open a bug and one of Drupal core developers answers: "Jean-Michel, this is a friendly warning, please change your behavior. This is getting really annoying." In fact, this core developer does not like the way I try to explain how to use databases. I wrote two tutorials: http://drupal.org/node/559302 and http://drupal.org/node/555514 The truth is that Drupal core developers do not believe fixing the prev/next link script is important. They don't care for SQL and don't understand the relationship between SQL queries and CPU cycles. Read this page: http://drupal.org/project/prev_next This performance issue was known for several months and these MySQL developers did not even fixed it. Finally, I escalated to the founder of the Drupal community to ask him to integrate a proper SQL Previous/Next script into Drupal. Their last answer: "Holy crap jmpoure, this is not how the community works. We don't beg to Dries." Read: http://drupal.org/node/559424 > Now, there are things which MySQL does better which we should fix, > because they are real problems for our users who already like > PostgreSQL. These include simple replication, upgrade in place, > driver > maintenance, covering indexes, MERGE, etc. But we'll do these because > they make *Postgres* better, not because MySQL has them. After reading my story, I hope we can agree that noone is going to port any MySQL code to PostgreSQL ever. This demands too much intellectual efforts. Many people will migrate from DB2 and Oracle to PostgreSQL. But no MySQL developer is going to use PostgreSQL if he needs to modify SQL queries. I don't want to be offensive, but I really believe it. So we should support a minimal set of MySQL SQL instructions. After several years of porting MySQL code to PostrgeSQL, I believe that this limited list is enough: http://drupal.org/node/555514 This is quite a straightforward need. Without this list of issues, PostgreSQL may never be able to run popular products developed under MySQL. Think of all commercial and free software projects. The impact of MySQLisms are huge. I can only compare it to Windows vs. GNU/Linux or FreeBSD. This is what comes in mind first. We are not leaving in a perfect world and there no reason to achieve perfectness. So let's support this list, please: http://drupal.org/node/555514 Kind regards, Jean-Michel
2009/8/26 Jean-Michel Pouré <jm@poure.com>: > After reading my story, I hope we can agree that noone is going to port > any MySQL code to PostgreSQL ever. This demands too much intellectual Surely this is a complete overgeneralization... ...Robert
Jean-Michel Pour� wrote: > To tell you how lazy MySQL people are is my last experience in the > Drupal world. In short, on my devel server, Drupal previous/next link > display SQL script returns 21.000 rows. Yes, we have seen this too. We have always targeted serious database developers, and those wanting to be serious. It is difficult to cater to MySQL eccentricities and maintain a serious database offering. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. +
Jean-Michel Pouré wrote: > We are not leaving in a perfect world and there no reason to achieve > perfectness. So let's support this list, please: > http://drupal.org/node/555514 Have you tried Drupal 7? It's said to have many of these corrected. Maybe you should stop wasting your time with 6.x. -- Alvaro Herrera http://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc.
2009/8/26 Jean-Michel Pouré <jm@poure.com>: > After reading my story, I hope we can agree that noone is going to port > any MySQL code to PostgreSQL ever. This demands too much intellectual > efforts. Many people will migrate from DB2 and Oracle to PostgreSQL. But > no MySQL developer is going to use PostgreSQL if he needs to modify SQL > queries. I don't want to be offensive, but I really believe it. > > So we should support a minimal set of MySQL SQL instructions. > > After several years of porting MySQL code to PostrgeSQL, I believe that > this limited list is enough: http://drupal.org/node/555514 > > This is quite a straightforward need. Without this list of issues, > PostgreSQL may never be able to run popular products developed under > MySQL. Think of all commercial and free software projects. The impact of > MySQLisms are huge. I can only compare it to Windows vs. GNU/Linux or > FreeBSD. This is what comes in mind first. > > We are not leaving in a perfect world and there no reason to achieve > perfectness. So let's support this list, please: > http://drupal.org/node/555514 > I'm starting to see how you rubbed the Drupal people the wrong way...... Most of the items on your list are misdiagnosed problems or are features that we think are bugs in MySQL. You would get further asking questions and for suggestions than you would insisting that you know better and Postgres should change its interface to match MySQL. With your current approach you're likely to get dismissed out of hand, not unlike what I can well believe happened in the Drupal world. That would be unfortunate because I think there are 2 or 3 real improvements hidden in your list. -- greg http://mit.edu/~gsstark/resume.pdf
On ons, 2009-08-26 at 14:26 -0400, Robert Haas wrote: > Sure, but an aimless mandate to do testing for 4 (or 8, or 12) months > doesn't necessarily buy you much, either. I'm good at focused > activity - but there was nothing focused about 8.4 beta that I could > see. Maybe we need some kind of TestFest process. Yeah, exactly. I can't imagine end users would know what to do during beta. Even assuming that you have release notes at the beginning of beta, you can't expect people to go through every item and do a formal test for it. Surely it's been tested before, else it would not be in the release, right?
Le mercredi 26 août 2009 à 22:06 +0100, Greg Stark a écrit : > With your current approach you're likely to get dismissed out of hand, > not unlike what I can well believe happened in the Drupal world. This is the case. > That > would be unfortunate because I think there are 2 or 3 real > improvements hidden in your list. Then explain I don't have your skills. Thanks.
Le mercredi 26 août 2009 à 16:56 -0400, Alvaro Herrera a écrit : > Have you tried Drupal 7? It's said to have many of these corrected. > Maybe you should stop wasting your time with 6.x. I am running a large community on the Net and people would like to migrate our framework to Drupal. We agreed to. I already assembled 30 modules to develop a complete service. But it does not scale very well as my testings show that queries are slow. I wrote SQL guides to inform the Drupal community. It seems that Drupal developers do not make a relationship between database and CPU time or memory usage. Therefore Drupal PHP cache is filled with SQL queries. At it demands more and more memory. I am probably going to migrate to Drupal 6.x in a few days and I will fix queries by hand in case of problems. This is what happened when we migrated to PhpBB. Bye, Jean-Michel
Tom, all, > As far as the alpha releases go, I wouldn't --- I see no evidence that > we have the manpower to formalize them any more than they are now. > I do like the idea of trying to reach out to more beta testers and > manage that phase more aggressively. Maybe if we can make something > happen there, it would be possible to do the same for alphas later. Well, we need a concrete list of things to beta test if we want people to do this in any organized fashion. It's easy enough for us to say "we need beta testers"; but without telling our community *what* to test, people just download, compile, do a select * from table, and say "it works". Once we have a list, we can launch a simple app/wiki page where people can report results of their tests. Without all that, the task is too amorphous for us to expect much from the community-at-large. -- Josh Berkus PostgreSQL Experts Inc. www.pgexperts.com
Josh Berkus wrote: > Tom, all, > > >> As far as the alpha releases go, I wouldn't --- I see no evidence that >> we have the manpower to formalize them any more than they are now. >> I do like the idea of trying to reach out to more beta testers and >> manage that phase more aggressively. Maybe if we can make something >> happen there, it would be possible to do the same for alphas later. >> > > Well, we need a concrete list of things to beta test if we want people > to do this in any organized fashion. It's easy enough for us to say "we > need beta testers"; but without telling our community *what* to test, > people just download, compile, do a select * from table, and say "it works". > > Once we have a list, we can launch a simple app/wiki page where people > can report results of their tests. > > Without all that, the task is too amorphous for us to expect much from > the community-at-large. > > Actually, what I had in mind was getting people to run their applications etc. in some non-production environment on the beta. I talked to a client today and he said "sure, we have several development environments and we can put one or two on the beta and then let the developers just do their thing on it." Testing the things we know about is in a way less important than making sure nothing else got broken. cheers andrew
On Wed, Aug 26, 2009 at 5:12 PM, Peter Eisentraut<peter_e@gmx.net> wrote: > On ons, 2009-08-26 at 14:26 -0400, Robert Haas wrote: >> Sure, but an aimless mandate to do testing for 4 (or 8, or 12) months >> doesn't necessarily buy you much, either. I'm good at focused >> activity - but there was nothing focused about 8.4 beta that I could >> see. Maybe we need some kind of TestFest process. > > Yeah, exactly. I can't imagine end users would know what to do during > beta. Even assuming that you have release notes at the beginning of > beta, you can't expect people to go through every item and do a formal > test for it. Surely it's been tested before, else it would not be in > the release, right? I would sure hope so. Testing features individually makes a whole lot more sense to me than testing the release as a whole. Just trying a bunch of random stuff and seeing if anything breaks is not a very productive activity. On the other hand, testing individual features is frequently very productive, but it's my understanding of the way PG does things that that is supposed to happen before the patch is committed. It appears to me that most of the really nasty bugs that have been found in 8.4.0 relate to one of the following three things, each of which seems to be related to multiple back-branch commits. 1. SEMI/ANTI join support. 2. running the bgwriter during recovery (infrastructure changes for recovery) 3. deadman switch Maybe some of these weren't tested well enough prior to commit? Or perhaps they're just more significant changes and therefore likely spots for rough edges. I think there is a lot of merit (as Andrew suggests) in running a production application on a beta version of the database just to see if anything funny happens. But insisting that all PG developers stop doing development to focus ONLY on that activity doesn't seem very reasonable: not everyone is well-placed to do that kind of experiment, or cares to do so. Conversely, there are many people who are NOT developers who ARE well-placed to beta test (for example, Kevin Grittner does a lot more testing than he does development, and I think there are few people on this mailing list who would argue that the quality of that testing is any less than kickass). ...Robert
Hi, Peter Eisentraut <peter_e@gmx.net> writes: > On ons, 2009-08-26 at 14:26 -0400, Robert Haas wrote: >> Sure, but an aimless mandate to do testing for 4 (or 8, or 12) months >> doesn't necessarily buy you much, either. I'm good at focused >> activity - but there was nothing focused about 8.4 beta that I could >> see. Maybe we need some kind of TestFest process. > > Yeah, exactly. I can't imagine end users would know what to do during > beta. Even assuming that you have release notes at the beginning of > beta, you can't expect people to go through every item and do a formal > test for it. Surely it's been tested before, else it would not be in > the release, right? Well we all know that developpers are really bad at testing, because they tend to test for cases they though about while developping the code rather than being creative and running a full application against the overall new product. Now, it could be that what we miss is some tool suite to enable application people to test their full code against our new releases and check results, performance, plans, etc. I know about a couple of tools to get there, Tsung and Playr. And the focus is performance testing and scalability, not that it still works. Is the offering good enough? We might need to run some kind of tutorials for users to be able to run large tests easily, and maybe think about some newer tools allowing to compare logs of two application runs in two database versions (capture all queries and result in a database, then have a way to diff). Then beta testing would mean having a spare machine where to run the magic regression test suite against some internal application. Regards, -- dim Tsung: http://tsung.erlang-projects.org/ Playr: https://area51.myyearbook.com/trac.cgi/wiki/Playr PS: sql level unit testing isn't an answer here as it means the application already have the tests when entering beta. Hard sell.
Robert Haas <robertmhaas@gmail.com> writes:
> On Wed, Aug 26, 2009 at 5:12 PM, Peter Eisentraut<peter_e@gmx.net> wrote:
>> ... Surely it's been tested before, else it would not be in
>> the release, right?
> I would sure hope so. Testing features individually makes a whole lot
> more sense to me than testing the release as a whole. Just trying a
> bunch of random stuff and seeing if anything breaks is not a very
> productive activity.
I beg to disagree. New features have presumably been tested, in
isolation, by the original developer and reviewers. The things we tend
to miss are bad side-effects on corner cases and seemingly-unrelated
features. So "testing as a whole" is exactly what beta is for, to my
mind.
> I think there is a lot of merit (as Andrew suggests) in running a
> production application on a beta version of the database just to see
> if anything funny happens.
... but here we seem to be coming out at the same place anyway. Getting
people to put their existing apps onto a beta is very productive.
We have to encourage people to do more of that while it's still beta,
instead of waiting till .0 or .1 or later.
regards, tom lane
2009/8/26 Jean-Michel Pouré <jm@poure.com>: > Le mercredi 26 août 2009 à 22:06 +0100, Greg Stark a écrit : >> That >> would be unfortunate because I think there are 2 or 3 real >> improvements hidden in your list. > > Then explain I don't have your skills. What I'm suggesting is that you should take a different direction, both with Drupal and with us. Instead of saying Postgres is wrong because of your analysis, ask why what's going wrong and what the right way to fix it is. So instead of telling people what to do you're asking for their help. People are a lot more friendly when asked for help than when being bossed around. Your list: > PostgreSQL and MySQL do not use the same concatenation funtions (D6 only, fixed in D7) Personally I don't see a problem with us adding this to Postgres now that we have variadic functions. I'm not sure why others are so dead-set against it; it seems a lot less burdensome than some of the Oracle compatibility stuff we have. > PostgreSQL does CAST implicitely between int and a domain derived from int (unsolved) This is misdiagnosed. It's not clear where the problem lies if there's a problem but it likely has nothing to do with the cast. If you posted with information and asked for help people would suggest things to test to get more information and eventually find the best way to fix it. > PostgreSQL does not allow DELETE on JOINS > PostgreSQL does not allow UPDATE or DELETE on multiple tables (not solved) These are a open issues. I would like to have better support here but it's a major feature, not just a minor compatibility issue. Again if you posted the queries and asked for suggestions people would help you rewrite the queries. > PostgreSQL does not allow mutiple ROW inserts (PostgreSQL 8.2 + only) As you point out this was added to Postgres 3 years ago > PostgreSQL does not allow nested ORDER BY I don't think this is something we're interested in doing. The grammar is hard enough to maintain as it is. > PostgreSQL does not automatically cast data between BOOLEAN and INT > PostgreSQL does not automatically cast data between INT and VARCHAR/CHAR These are things we've gone out of our way to NOT do. At some cost too. Being loose here makes it easy to miss errors in your SQL. > PostgreSQL requires all non-aggregated fields to be present in the GROUP BY clause As I explained at length we could do something here but it would be a major feature and it would still not be compatible with MySQL unless you happen to be using it in a particular (common) way. For more general uses you have to use DISTINCT ON. > PostgreSQL requires single quotes and not double-quotes There's no way in the world we would switch this. We're an SQL engine and we parse the SQL language which uses quotes in a certain way. If we switched this we would be parsing some other very different language and everything else written in SQL (including, say, internal queries for referential integrity checks or psql tab completion or pg_dump) would break. -- greg http://mit.edu/~gsstark/resume.pdf
2009/8/26 Greg Stark <gsstark@mit.edu>: >> PostgreSQL does not automatically cast data between BOOLEAN and INT >> PostgreSQL does not automatically cast data between INT and VARCHAR/CHAR > > These are things we've gone out of our way to NOT do. At some cost > too. Being loose here makes it easy to miss errors in your SQL. Actually it always bothered me that we don't have implicit casts from integer->boolean. I can't see any ambiguity or unintentional effects this would cause problems with. Am I missing something? -- greg http://mit.edu/~gsstark/resume.pdf
Greg Stark wrote: > > PostgreSQL and MySQL do not use the same concatenation funtions (D6 only, fixed in D7) > > Personally I don't see a problem with us adding this to Postgres now > that we have variadic functions. I'm not sure why others are so > dead-set against it; it seems a lot less burdensome than some of the > Oracle compatibility stuff we have. The Oracle functions are there to add functionality that can't be easily performed using ANSI syntax; this is not true of concact() and ||. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. +
Greg Stark <gsstark@mit.edu> wrote: > Actually it always bothered me that we don't have implicit casts > from integer->boolean. I can't see any ambiguity or unintentional > effects this would cause problems with. Am I missing something? I'd be at least a little bit concerned about how such automatic casting to boolean might interact with bug #3822: http://archives.postgresql.org/pgsql-bugs/2007-12/msg00145.php -Kevin
On Wed, Aug 26, 2009 at 11:15 PM, Tom Lane<tgl@sss.pgh.pa.us> wrote: > ... but here we seem to be coming out at the same place anyway. Getting > people to put their existing apps onto a beta is very productive. > We have to encourage people to do more of that while it's still beta, > instead of waiting till .0 or .1 or later. +1 for the goal being to get users to test their applications on beta. I also wonder if we've adopted the wrong strategy with betas by stopping development during them. It seems to be the worst of both worlds in that both developers and users are unhappy. Perhaps we should fork 8.6 right away after 8.5.beta is released (I'm assuming *after* any open issues are closed) and start a commitfest with any pending patches. While we do that call for users to test 8.5.beta with their applications and wait a fixed amount of time for any bugs to turn up. That lets us have a long beta period for users to test on without stopping development. -- greg http://mit.edu/~gsstark/resume.pdf
On Thu, Aug 27, 2009 at 12:03 AM, Dimitri Fontaine<dfontaine@hi-media.com> wrote: > Is the offering good enough? We might need to run some kind of tutorials > for users to be able to run large tests easily, and maybe think about > some newer tools allowing to compare logs of two application runs in two > database versions (capture all queries and result in a database, then > have a way to diff). Then beta testing would mean having a spare machine > where to run the magic regression test suite against some internal > application. Well, people may recall I spent a lot of time testing 8.3 before and during beta. The important words are "a lot of time", probably one month full time spread on 3 months to find *only* 3 problems Tom, Alvarro and Andrew fixed: yes one month for only 3 problems identified, reported, discussed and fixed. The problem isn't to connect your application to the database - it's the easy part: if you have a large one, you probably won't see the anomalies. The application I used for my tests is displaying every SQL query at the bottom of the page with the time spent executing the query, I was switching from the 8.1 site to the 8.3 site to check everything manually. I also got all the urls of this application (more than one million), and use a load test tool to load each page and pgFouine to grab any error from the PostgreSQL logs. Even with these information and this work, I'm pretty sure I would have missed a join problem which would have returned 2 or 3 more rows even with the time I spent working on it. My plan at the time was to develop an application which would parse the query logs from the server, replay the queries on 2 PG servers with both versions and report me the anomalies (difference in the number of rows, difference in the content, queries slower than with the old version). I haven't had the opportunity to work on the 8.4 beta test due to my daily (and often nightly) job work load but the idea is still there and IMHO it's really necessary if we want to be able to detect the problems we only discovered after the release. -- Guillaume
Jean-Michel, > The truth is that Drupal core developers do not believe fixing the > prev/next link script is important. They don't care for SQL and don't > understand the relationship between SQL queries and CPU cycles. I doubt this would be an accurate description of all Drupal developers.The Drupal developers I've met have been very intelligent;most don't know much about SQL database, but then I know zero JavaScript so we're even. The answer to this is to be educational and not confrontational. And submit patches ... to a current development version and not an old version. > After reading my story, I hope we can agree that noone is going to port > any MySQL code to PostgreSQL ever. Again, the evidence is against you. > So we should support a minimal set of MySQL SQL instructions. In a loadable module, sure. Patches? > After several years of porting MySQL code to PostrgeSQL, I believe that > this limited list is enough: http://drupal.org/node/555514 There's other stuff as well. Also, see the updates people have made to your list. > can only compare it to Windows vs. GNU/Linux or > FreeBSD. This is what comes in mind first. Well, I don't use Windows either. So that's a pretty weak argument with me ... -- Josh Berkus PostgreSQL Experts Inc. www.pgexperts.com
On ons, 2009-08-26 at 18:15 -0400, Tom Lane wrote: > > I think there is a lot of merit (as Andrew suggests) in running a > > production application on a beta version of the database just to see > > if anything funny happens. > > ... but here we seem to be coming out at the same place anyway. Getting > people to put their existing apps onto a beta is very productive. > We have to encourage people to do more of that while it's still beta, > instead of waiting till .0 or .1 or later. I think people should be running their applications' system tests on top of the new PostgreSQL. Just installing the application, clicking three buttons, and I-don't-have-more-time-than-this helps a little but not much. Of course many people won't have system tests, which is why this process is a problem. To pick up a current example: "Drupal system tests pass with 8.5betaN" is nice and useful. "I ran our app on 8.5betaN and didn't see any issues" is interesting, but ultimately doesn't help much. Much of the delay and uncertainty during beta in my mind comes from the situation that we wait for negative results and don't trust the release until we have seen and fixed enough of them. Instead of waiting for concrete, positive results and producing the release with confidence.
Peter Eisentraut wrote: > Much of the delay and uncertainty during beta in my mind comes from the > situation that we wait for negative results and don't trust the release > until we have seen and fixed enough of them. Instead of waiting for > concrete, positive results and producing the release with confidence. Yep, that is our dilemma. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. +
On Aug 26, 2009, at 8:17 AM, Jean-Michel Pouré wrote: > Le mercredi 26 août 2009 à 01:36 -0600, Rick Gigger a écrit : >> One possible reason that replication is more critical now than it >> was >> a year ago is the rise in cloud computing. The ability to fire up >> instances on demand is much more useful when some of those boxes can >> be database servers and those databases servers can replicate the >> primary database and start doing something useful. As far as I can >> tell this one feature alone is the one thing that makes it hard to >> convince people to migrate away from Mysql despite it's demonstrable >> inferiority in many other areas. > > I think you should have a deep look at > these two manuals that I wrote for Drupal: > > Guidelines for writing MySQL and PostgreSQL compliant SQL > http://drupal.org/node/555514 > > and > > Quidelines for writing SQL efficient code: > http://drupal.org/node/559302 > > I have been using PostgreSQL since 1998. I took part in the > development > of pgAdmin 1 and pgAdmin 2. I am very proud of the PostgreSQL > community, > but I think it should fix some important issues in the domain of SQL > compliance and compatibility. > > When reading developers comments on Drupal web site, it seems that > there > is deep misunderstanding between developers and SQL databases. I would > say that 1% of developers know database technology. For example, most > Drupal developers think that an INNER JOIN is faster than a LEFT JOIN. > > The reality of facts is that developers will not even test PostgreSQL > and stop using it after the first SQL warning or error. > > So I would recommend focussing on usability. > > Then you can work on replication and materilized views. You probably > know that a cloud requires several computers. With materialized > view, a > single computer can perform 100 times better. I see plenty of of > possibilities to improve speed using materialized views. > > But first and firstly ... focus on usability. Then make a pre- > release of > a PostgreSQL 8.4.2 release or so ... We need to wipe out this MySQL > issue once for all. > > If there is a compat MySQL mode or functions, then include it in core. > This is too important for PostgreSQL success. > > Why MySQL usability is achieved add materialized views and MySQL is > dead > in the performance statistics, beaten 99% by PostgreSQL. This may be your experience and maybe there are stats to back this up. I was simply saying, that in my experience I have consulted with companies using cloud computing services (ie EC2) and mysql. They are performance conscious. When they have to fire up more boxes, they pay for it immediately. When they ran into problems getting good performance out of mysql it was very easy to show them how to get better performance using postgres. (Often this was just: do the same thing in postgres, and look, it's faster!). But they also rely on replication to be able to scale. And without it they just weren't interested. Porting any project has it's own set of issues, I was speaking to the case where people are evaluating databases for a new project. I was not however trying to make any kind of statement as too how important it is as compared to any other specific feature. I was just trying to say that in my experience current trends indicate that having easy to set up replication is getting more important over time, not less, and not the same. Other features may be more important. Getting it right is certainly more important that getting it soon (for reasonable values of "soon" and "right" of course IMHO). The experience of others my be totally different, but that is mine. Just wanted to clarify what I was actually trying to say because this response seems to indicate that I didn't make certain things clear. - Rick
Greg Stark <gsstark@mit.edu> writes:
> Actually it always bothered me that we don't have implicit casts from
> integer->boolean. I can't see any ambiguity or unintentional effects
> this would cause problems with. Am I missing something?
Personally, as an old Pascal-lover, I always thought that C's failure
to distinguish between int and boolean was the single biggest design
mistake in the language. I'm very glad that SQL doesn't make that
mistake, and I don't want to go against the standard to introduce it.
regards, tom lane
* Andrew Dunstan (andrew@dunslane.net) wrote:
> Actually, what I had in mind was getting people to run their
> applications etc. in some non-production environment on the beta. I
> talked to a client today and he said "sure, we have several development
> environments and we can put one or two on the beta and then let the
> developers just do their thing on it." Testing the things we know about
> is in a way less important than making sure nothing else got broken.
I agree entirely with Andrew here- what we need are a set of users who
would be willing to run their actual applications against a beta release
in a testing environment. The Beta-Mom position would be working with
some list of users who've volunteered to do that; prodding them when a
new beta comes out, poking them for feedback, working with them on
issues they run into, etc.
The other possible group of users are those who are interested and
willing to beta-test actual new external-facing features. That'd be
great to have as well, but I don't believe is as important. New
features having bugs are a much smaller impact, overall, than bugs which
have been introduced in existing code-paths due to changes.
Thanks,
Stephen
On Thu, Aug 27, 2009 at 1:24 AM, Tom Lane<tgl@sss.pgh.pa.us> wrote: > Greg Stark <gsstark@mit.edu> writes: >> Actually it always bothered me that we don't have implicit casts from >> integer->boolean. I can't see any ambiguity or unintentional effects >> this would cause problems with. Am I missing something? > > Personally, as an old Pascal-lover, I always thought that C's failure > to distinguish between int and boolean was the single biggest design > mistake in the language. I'm very glad that SQL doesn't make that > mistake, and I don't want to go against the standard to introduce it. I'm sure I can think of bigger flaws in C than that :) I tend to think SQL has more in common with lisp than either of those, perhaps because of the tinge of functional programming style. But I think you're generalizing my suggestion to the point of building a straw man to say "failure to distinguish". You could argue that using a boolean in places where integers are expected could be confusing or dangerous. But using other data types where boolean values are expected is perfectly reasonable and safe -- especially in cases like integer where people do expect it to work and the behaviour is very predictable. -- greg http://mit.edu/~gsstark/resume.pdf
Tom Lane wrote: > Josh Berkus <josh@agliodbs.com> writes: >>> That is a slightly alarmist. Who are we going to lose these users to? > >> Drizzle. MySQL forks. CouchDB. Any database which has replication >> which you don't need a professional DBA to understand. Whether or not >> it works. > > You haven't explained why we'd lose such folk next year when we haven't > lost them already. MySQL has had replication (or at least has checked > off the bullet point ;-)) for years. I think it's a slow but ongoing stream of organizations that are switching away using logic similar to the thoughts outlined here: http://archives.postgresql.org/pgsql-hackers/2008-05/msg00955.php "...switched their bugzilla from Postgres to MySQL because theadmins didn't want to deal with Slony any more. People wantsimple." MySQL may not have caught postgres in a number of ways yet, but it's good enough now for many of the things it wasn't good enough for earlier. And if it's good enough and easier, it's easy to switch.
On Thu, Aug 27, 2009 at 2:30 AM, Stephen Frost<sfrost@snowman.net> wrote: > I agree entirely with Andrew here- what we need are a set of users who > would be willing to run their actual applications against a beta release > in a testing environment. The Beta-Mom position would be working with > some list of users who've volunteered to do that; prodding them when a > new beta comes out, poking them for feedback, working with them on > issues they run into, etc. I don't think we need the Beta-Mom to work with them on issues they run into. We want to hear those things on the mailing lists. But I do think the idea of having someone being Beta-Mom is a good idea because of this simple point: right now we don't know if we have any beta testers. All we hear about is when they run into problems. What we want to know is how many people have run their applications and *not* had problems. So I see the Beta-Mom as being in charge of: Gathering a list of volunteers willing to report results For each volunteer getting a list of features they tested For each volunteer getting a "yay" or "nay" report Ideally we would want to know that people have run their application frameworks for a certain length of time, with slave databases set up, with slony set up, tested failover, with various pluggable langauges, with various contrib modules, with large objects, with full text search, etc. Once we have a reasonable number of positive reports covering a wide range of heavy-duty configuration then we can start to base our confidence with going ahead with a release on those positive reports rather than simply on the amount of time that's passed without a problem. -- greg http://mit.edu/~gsstark/resume.pdf
Tom Lane wrote: > Greg Stark <gsstark@mit.edu> writes: >> Actually it always bothered me that we don't have implicit casts from >> integer->boolean. I can't see any ambiguity or unintentional effects >> this would cause problems with. Am I missing something? > > Personally, as an old Pascal-lover, I always thought that C's failure > to distinguish between int and boolean was the single biggest design > mistake in the language. I'm very glad that SQL doesn't make that > mistake, and I don't want to go against the standard to introduce it. Then you should love Python where everything non-empty is regarded True in boolean context ;-) Cheers Tino
Re: MySQL Compatibility WAS: postgresql.conf settings autoconfiguration
From
Jean-Michel Pouré
Date:
Dear all, Just a quick message about postgresql.conf auto-configuration. When MySQL users test PostgreSQL, they load their data and run simple SQL queries. If postgresql.conf is configured with default values, it may produce slow results. Would there be a way for postgresql.conf to auto configure? This is how MySQL works. Nothing is configurable. And people love it. So why not implement a simple auto configuration mechanism which would set basic parameters. Example autotune=true; autotune_profile='web site'; autotune_profile='dedicated SQL server'; ... Kind regards, Jean-Michel
Tom Lane írta: > Greg Stark <gsstark@mit.edu> writes: > >> Actually it always bothered me that we don't have implicit casts from >> integer->boolean. I can't see any ambiguity or unintentional effects >> this would cause problems with. Am I missing something? >> > > Personally, as an old Pascal-lover, I always thought that C's failure > to distinguish between int and boolean was the single biggest design > mistake in the language. On the other hand, the first two programming languages I have learnt was Sinclair ZX Spectrum BASIC and Z80 assembly, I welcomed C being so near to assembly... :-) > I'm very glad that SQL doesn't make that > mistake, and I don't want to go against the standard to introduce it. > In that we agree. SQL is not C. Best regards, Zoltán Böszörményi -- Bible has answers for everything. Proof: "But let your communication be, Yea, yea; Nay, nay: for whatsoever is more than these cometh of evil." (Matthew 5:37) - basics of digital technology. "May your kingdom come" - superficial description of plate tectonics ---------------------------------- Zoltán Böszörményi Cybertec Schönig & Schönig GmbH http://www.postgresql.at/
On Wed, Aug 26, 2009 at 7:39 PM, Bruce Momjian<bruce@momjian.us> wrote: > Peter Eisentraut wrote: >> Much of the delay and uncertainty during beta in my mind comes from the >> situation that we wait for negative results and don't trust the release >> until we have seen and fixed enough of them. Instead of waiting for >> concrete, positive results and producing the release with confidence. > > Yep, that is our dilemma. To get positive results in which you can have confidence, you have to know that the testing which was done actually did a reasonably good job exercising the code in a way that would have flushed out bugs, had any been present. That sounds a lot like the definition of a regression test suite. Of course, we have that already, but it's nowhere near comprehensive. Maybe we should be looking at an expanded test suite that runs on a time scale of hours rather than seconds. Actually, didn't Peter talk about something like this at PGCon? I don't think that any test suite is going to eliminate the need for beta-testing. But if we could say that we had a regression test suite which covered X% of our code, and it passed on all Y platforms tested, that would certainly be a confidence booster, especially for large values of X. What we're basically doing now is hoping that beta-testers come up with some novel tests, and that's a bit hit-or-miss. Part of the question, of course, is how to build up such a regression test suite. One way to do it is try to add a test every time you fix a bug, such that the new test fails on the unpatched code and passes with the fix... or we could have a expanded-regression-test only commitfest during beta, or something. I think people would be willing to put in some work on this. I certainly would. I run the regression tests frequently during development but unless I'm making system catalog changes they rarely catch anything. I'm not prepared to write the whole thing myself, but I would definitely be willing to develop and contribute some tests. ...Robert
Robert Haas <robertmhaas@gmail.com> wrote: > Maybe we should be looking at an expanded test suite that runs on a > time scale of hours rather than seconds. > if we could say that we had a regression test suite which covered X% > of our code, and it passed on all Y platforms tested, that would > certainly be a confidence booster, especially for large values of X. > Part of the question, of course, is how to build up such a > regression test suite. Aren't there code coverage monitoring tools that could be run during regression tests? Sure it would take some time to review the results and fashion tests to exercise chunks of code which were missed, but at least we could quantify X and try to make incremental progress on increasing it.... -Kevin
Le mercredi 26 août 2009 à 15:51 -0700, Josh Berkus a écrit : > I doubt this would be an accurate description of all Drupal > developers. My opinion was : Before adding replication to PostgreSQL, it would be better to support a basic set of MySQL syntax seems relevant: DELETE FROM table1, table2 FROM table1 INNER JOIN table2 ON ... UPDATE table1, table1 SET table1.foo=1, table2.bar=3 WHERE table1.foo=2 INNER JOIN table2 ..., etc ... There is not much to add to PostgreSQL, but it seems relevant. Otherwise, replicating some MySQL SQL syntax will not work. As you know, people willing to use PostgreSQL replication are possibly already MySQL replication users. So if they test and PostgreSQL fails, this is too bad. Drupal was only an example, I did not mean to criticize all Drupal developers. Just to say they focus on PHP and not SQL. They don't have time to port MySQL code. Besides, they are very nice people. So I apologize for this short (false) focus. Bye, Jean-Michel
Robert Haas <robertmhaas@gmail.com> writes:
> ... That sounds a lot like the definition of a
> regression test suite. Of course, we have that already, but it's
> nowhere near comprehensive. Maybe we should be looking at an expanded
> test suite that runs on a time scale of hours rather than seconds.
mysql's got one of those, and I haven't noticed that it's kept their
defect rate down any. Hour-long regression suites are the sort of
thing developers won't run. Worse, regression suites are necessarily
designed to exercise only 100%-reproducible behavior.
> I don't think that any test suite is going to eliminate the need for
> beta-testing.
Precisely...
What I'd like to see is some sort of test mechanism for WAL recovery.
What I've done sometimes in the past (and recently had to fix the tests
to re-enable) is to kill -9 a backend immediately after running the
regression tests, let the system replay the WAL for the tests, and then
take a pg_dump and compare that to the dump gotten after a conventional
run. However this is quite haphazard since (a) the regression tests
aren't especially designed to exercise all of the WAL logic, and (b)
pg_dump might not show the effects of some problems, particularly not
corruption in non-system indexes. It would be worth the trouble to
create a more specific test methodology.
In short: merely making the tests bigger doesn't impress me in the
least. Focused testing on areas we aren't covering at all could be
worth the trouble.
regards, tom lane
On Thu, Aug 27, 2009 at 10:11 AM, Tom Lane<tgl@sss.pgh.pa.us> wrote: > Robert Haas <robertmhaas@gmail.com> writes: >> ... That sounds a lot like the definition of a >> regression test suite. Of course, we have that already, but it's >> nowhere near comprehensive. Maybe we should be looking at an expanded >> test suite that runs on a time scale of hours rather than seconds. > > mysql's got one of those, and I haven't noticed that it's kept their > defect rate down any. Hour-long regression suites are the sort of > thing developers won't run. Well, I'll run them. And I bet we could get volunteers to provide machines to run them every night, too, against CVS HEAD. This has been discussed before, and I wasn't the one who suggested it. I can't speak to the value (or lack thereof) of mysql's regression test suite as I know nothing about it, but even if it is completely worthless that does not prove that a worthwhile test suite can't be constructed. > Worse, regression suites are necessarily > designed to exercise only 100%-reproducible behavior. That is true, but our current testing methodology (hoping the beta-testers find the bugs) seems not to be completely satisfactory either. Which brings us to: >> I don't think that any test suite is going to eliminate the need for >> beta-testing. > > Precisely... > > What I'd like to see is some sort of test mechanism for WAL recovery. > What I've done sometimes in the past (and recently had to fix the tests > to re-enable) is to kill -9 a backend immediately after running the > regression tests, let the system replay the WAL for the tests, and then > take a pg_dump and compare that to the dump gotten after a conventional > run. However this is quite haphazard since (a) the regression tests > aren't especially designed to exercise all of the WAL logic, and (b) > pg_dump might not show the effects of some problems, particularly not > corruption in non-system indexes. It would be worth the trouble to > create a more specific test methodology. Yep. I was thinking about this as an area for possible improvement. I don't immediately have a brilliant idea how to do it. I did have the idea of creating a loadable C function which would panic the database. This could be used to crash the database at a particular point (even in mid-query, with sufficient creativity). I think we would certainly need some more powerful way of checking pass/failure than exact-text comparisons on SQL query results. Being a perl guy, the first thing that occurs to me is to write some kind of test harness in perl that can issue SQL queries as well as do other things, but I don't have an exact design mapped out in my head, and I'm sure there are other viable approaches. > In short: merely making the tests bigger doesn't impress me in the > least. Focused testing on areas we aren't covering at all could be > worth the trouble. Well, I wasn't suggesting adding a lot more testing of things that we're already testing. I was assuming that we would craft the additional tests to hit areas that we are not now covering well. My point here is only to what Peter said upthread: we want to be able to get positive results rather than waiting for "enough" negative results (whatever that means). To get positive results, you must have a test suite. While letting beta testers test whatever they want has some value, testing things we think might be likely hiding places for bugs (such as WAL recovery) has merit, too. Making those tests well-organized and easily repeatable is, IMHO, a Good Thing. ...Robert
Robert Haas <robertmhaas@gmail.com> writes:
> Well, I wasn't suggesting adding a lot more testing of things that
> we're already testing. I was assuming that we would craft the
> additional tests to hit areas that we are not now covering well. My
> point here is only to what Peter said upthread: we want to be able to
> get positive results rather than waiting for "enough" negative results
> (whatever that means). To get positive results, you must have a test
> suite. While letting beta testers test whatever they want has some
> value, testing things we think might be likely hiding places for bugs
> (such as WAL recovery) has merit, too. Making those tests
> well-organized and easily repeatable is, IMHO, a Good Thing.
The problem here is the "easily repeatable" bit. Almost by definition,
easily repeatable tests don't find hard-to-reproduce problems. I don't
mean to suggest that they're without value, but they are no substitute
for beta testers doing unpredictable things.
regards, tom lane
On Thu, Aug 27, 2009 at 3:11 PM, Tom Lane<tgl@sss.pgh.pa.us> wrote: > > Precisely... > > What I'd like to see is some sort of test mechanism for WAL recovery. > What I've done sometimes in the past (and recently had to fix the tests > to re-enable) is to kill -9 a backend immediately after running the > regression tests, let the system replay the WAL for the tests, and then > take a pg_dump and compare that to the dump gotten after a conventional > run. However this is quite haphazard since (a) the regression tests > aren't especially designed to exercise all of the WAL logic, and (b) > pg_dump might not show the effects of some problems, particularly not > corruption in non-system indexes. It would be worth the trouble to > create a more specific test methodology. What I've been thinking of doing is having the regression suite take a backup after initdb and set archive mode on. when the regression suite finishes start the backup up and replay all the WAL. I'm not sure how to compare the databases since I don't think pg_dump actually works here -- a lot of the data is dropped by the end of the test. But at least that would test that replaying WAL didn't cause any crashes. -- greg http://mit.edu/~gsstark/resume.pdf
On Thu, Aug 27, 2009 at 11:29:42AM -0400, Tom Lane wrote: > Robert Haas <robertmhaas@gmail.com> writes: > > Well, I wasn't suggesting adding a lot more testing of things that > > we're already testing. I was assuming that we would craft the > > additional tests to hit areas that we are not now covering well. My > > point here is only to what Peter said upthread: we want to be able to > > get positive results rather than waiting for "enough" negative results > > (whatever that means). To get positive results, you must have a test > > suite. While letting beta testers test whatever they want has some > > value, testing things we think might be likely hiding places for bugs > > (such as WAL recovery) has merit, too. Making those tests > > well-organized and easily repeatable is, IMHO, a Good Thing. > > The problem here is the "easily repeatable" bit. Almost by definition, > easily repeatable tests don't find hard-to-reproduce problems. I don't > mean to suggest that they're without value, but they are no substitute > for beta testers doing unpredictable things. I've wondered before about using a system emulator to snapshot the disk on each write (I'd expect you could put some pretty low level hooks in with qemu and gdb) and then run each snapshot in another system to make sure that either the transaction is rolled back or committed as appropriate. I guess this would take a while to run but may help catch some obscure bugs. Or this an area that's well tested already? -- Sam http://samason.me.uk/
On Thu, Aug 27, 2009 at 11:29 AM, Tom Lane<tgl@sss.pgh.pa.us> wrote: > Robert Haas <robertmhaas@gmail.com> writes: >> Well, I wasn't suggesting adding a lot more testing of things that >> we're already testing. I was assuming that we would craft the >> additional tests to hit areas that we are not now covering well. My >> point here is only to what Peter said upthread: we want to be able to >> get positive results rather than waiting for "enough" negative results >> (whatever that means). To get positive results, you must have a test >> suite. While letting beta testers test whatever they want has some >> value, testing things we think might be likely hiding places for bugs >> (such as WAL recovery) has merit, too. Making those tests >> well-organized and easily repeatable is, IMHO, a Good Thing. > > The problem here is the "easily repeatable" bit. Almost by definition, > easily repeatable tests don't find hard-to-reproduce problems. I don't > mean to suggest that they're without value, but they are no substitute > for beta testers doing unpredictable things. I think you're overstating the case, but I don't want to argue the point, particularly. What I do want to do is find a way to address the problem described in the last sentence of this email: http://archives.postgresql.org/pgsql-hackers/2009-08/msg01614.php Both committers and non-committers are currently suffering from the fact that there is not a lot of time in the year which is set aside for development, i.e. neither CommitFest-time nor beta-time. To fix this problem, we can: 1. Make CommitFests shorter. 2. Make CommitFests less frequent. 3. Continue developing during CommitFests. 4. Make beta cycles shorter. 5. Make beta cycles less frequent (i.e. lengthen the release cycle). 6. Continue developing during beta. I believe (1) to be completely impractical and (3) to be self-defeating. I suspect (2) will backfire badly. That doesn't leave us with a lot of options. We can certainly do (5), but the downside is that features that get committed won't hit release for a very long time. I and others have suggested a couple of possible approaches toward (4) or (6), such as changing the way we do release notes, adding more regression tests to give us more (not perfect) confidence that the release is solid, and/or branching the tree during beta. None of those ideas have gotten a single vote of confidence from you or Bruce. What's your suggestion? ...Robert
On Thu, Aug 27, 2009 at 10:38 AM, Greg Stark<gsstark@mit.edu> wrote: > > What I've been thinking of doing is having the regression suite take a > backup after initdb and set archive mode on. when the regression suite > finishes start the backup up and replay all the WAL. > > I'm not sure how to compare the databases - execute 60 of the 121 tests (or at least those that create tables and insert/update/delete the most data) - crash the server and replay the WAL - execute the rest of the tests and cross your fingers :) -- Atentamente, Jaime Casanova Soporte y capacitación de PostgreSQL Asesoría y desarrollo de sistemas Guayaquil - Ecuador Cel. +59387171157
Greg Stark <gsstark@mit.edu> writes:
> On Thu, Aug 27, 2009 at 3:11 PM, Tom Lane<tgl@sss.pgh.pa.us> wrote:
>> ... However this is quite haphazard since (a) the regression tests
>> aren't especially designed to exercise all of the WAL logic, and (b)
>> pg_dump might not show the effects of some problems, particularly not
>> corruption in non-system indexes. �It would be worth the trouble to
>> create a more specific test methodology.
> What I've been thinking of doing is having the regression suite take a
> backup after initdb and set archive mode on. when the regression suite
> finishes start the backup up and replay all the WAL.
> I'm not sure how to compare the databases since I don't think pg_dump
> actually works here -- a lot of the data is dropped by the end of the
> test.
Yeah, that's another problem with using the existing tests for this
purpose --- a lot of possibly-useful stuff isn't kept around to the end.
And the desire to keep the test modules independent limits the amount of
interaction between them too. I really think we'd need a bespoke set of
tests to get very far with this.
This reminds me that pg_dump/pg_restore is another large pile of code
that receives no formalized testing whatsoever ...
regards, tom lane
2009/8/27 Jean-Michel Pouré <jm@poure.com>: > > Otherwise, replicating some MySQL SQL syntax will not work. > > As you know, people willing to use PostgreSQL replication are possibly > already MySQL replication users. So if they test and PostgreSQL fails, > this is too bad. > yeah! but some times the reason MySQL is not failing on those cases is because it's broken... Consider this one: http://archives.postgresql.org/pgsql-general/2005-12/msg00487.php we don't want to copy the bugs nor the bad designs decisions from MySQL, no that everything from MySQL is bad but i would be scary if we start supporting every single piece of code MySQL accepts -- Atentamente, Jaime Casanova Soporte y capacitación de PostgreSQL Asesoría y desarrollo de sistemas Guayaquil - Ecuador Cel. +59387171157
2009/8/27 Jaime Casanova <jcasanov@systemguards.com.ec>: > 2009/8/27 Jean-Michel Pouré <jm@poure.com>: >> >> Otherwise, replicating some MySQL SQL syntax will not work. >> >> As you know, people willing to use PostgreSQL replication are possibly >> already MySQL replication users. So if they test and PostgreSQL fails, >> this is too bad. >> > > yeah! but some times the reason MySQL is not failing on those cases is > because it's broken... > Consider this one: > http://archives.postgresql.org/pgsql-general/2005-12/msg00487.php > > we don't want to copy the bugs nor the bad designs decisions from > MySQL, no that everything from MySQL is bad but i would be scary if we > start supporting every single piece of code MySQL accepts > And that behavior has changed to be sane in 5.0+, iirc. -- Rob Wultsch wultsch@gmail.com
On tor, 2009-08-27 at 09:58 -0400, Robert Haas wrote: > To get positive results in which you can have confidence, you have to > know that the testing which was done actually did a reasonably good > job exercising the code in a way that would have flushed out bugs, had > any been present. That sounds a lot like the definition of a > regression test suite. Of course, we have that already, but it's > nowhere near comprehensive. Maybe we should be looking at an expanded > test suite that runs on a time scale of hours rather than seconds. > Actually, didn't Peter talk about something like this at PGCon? Let's look at it this way: If I were writing a compiler, then I would have two main test approaches. First, I would have an in-tree test suite that compiles a bunch of example code snippets and checks that the results are reasonable. Call that a regression test. It would be informed by code coverage analysis and previously reported bugs. Second, as part of my release cycle, I would have an agenda to try to compile a large set of real programs against my new compiler version. I would do that during the beta period. You will notice that GCC pretty much operates that way. We have regression tests. They could and should be expanded. That's a developer job, and we can start working on that now. But this discussion was about what to do during beta. And I think during beta you want to test PostgreSQL against a large set of real applications. But we could try to clarify how to actually do that in an organized way. Now, if you want to improve the regression tests, I would suggest going through the commits since 8.4beta and since 8.4.0 final release and ask how these problems could have been prevented or caught earlier. I suppose a test suite for WAL might be part of the answer, but a closer analysis might be insightful.
On Thu, Aug 27, 2009 at 2:54 PM, Peter Eisentraut<peter_e@gmx.net> wrote: > On tor, 2009-08-27 at 09:58 -0400, Robert Haas wrote: >> To get positive results in which you can have confidence, you have to >> know that the testing which was done actually did a reasonably good >> job exercising the code in a way that would have flushed out bugs, had >> any been present. That sounds a lot like the definition of a >> regression test suite. Of course, we have that already, but it's >> nowhere near comprehensive. Maybe we should be looking at an expanded >> test suite that runs on a time scale of hours rather than seconds. >> Actually, didn't Peter talk about something like this at PGCon? > > Let's look at it this way: If I were writing a compiler, then I would > have two main test approaches. First, I would have an in-tree test > suite that compiles a bunch of example code snippets and checks that the > results are reasonable. Call that a regression test. It would be > informed by code coverage analysis and previously reported bugs. > Second, as part of my release cycle, I would have an agenda to try to > compile a large set of real programs against my new compiler version. > I would do that during the beta period. You will notice that GCC pretty > much operates that way. > > We have regression tests. They could and should be expanded. That's a > developer job, and we can start working on that now. But this > discussion was about what to do during beta. And I think during beta > you want to test PostgreSQL against a large set of real applications. > But we could try to clarify how to actually do that in an organized way. > > Now, if you want to improve the regression tests, I would suggest going > through the commits since 8.4beta and since 8.4.0 final release and ask > how these problems could have been prevented or caught earlier. I > suppose a test suite for WAL might be part of the answer, but a closer > analysis might be insightful. What I want to do is address the concern about too much of any given year being consumed by beta and CommitFest. I'm not sure I know how to do that though. ...Robert
2009/8/27 Rob Wultsch <wultsch@gmail.com>: > > And that behavior has changed to be sane in 5.0+, iirc. > 5.0.12+ actually... that is stated in the same thread... the point was that if we simply were saying: hey! mysql can interpret this, make postgres do the same then we could end up with a lot of broken stuff... just because mysql users think is wonderful to not have to write sane code... -- Atentamente, Jaime Casanova Soporte y capacitación de PostgreSQL Asesoría y desarrollo de sistemas Guayaquil - Ecuador Cel. +59387171157
Hi, Robert Haas <robertmhaas@gmail.com> writes: > On Thu, Aug 27, 2009 at 2:54 PM, Peter Eisentraut<peter_e@gmx.net> wrote: >> We have regression tests. They could and should be expanded. That's a >> developer job, and we can start working on that now. But this >> discussion was about what to do during beta. And I think during beta >> you want to test PostgreSQL against a large set of real applications. >> But we could try to clarify how to actually do that in an organized >> way. Exactly, and I think that what we're missing here is a simple tool for our users to check a new PostgreSQL release against their existing application. We already know how to either log all queries and analyze the log files (CSV makes it easier, pgfouine parses them too) or to have a fe/be protocol proxy to record application SQL traffic (tsung recorder does that). What we miss is a tool to run the captured queries through both versions of PG and report any resultset mismatch, of course with a way to account for ordering issues (but we've seen people rely on the ordering when they don't give an order by clause, then bug the lists about it if a new release changes it). > What I want to do is address the concern about too much of any given > year being consumed by beta and CommitFest. I'm not sure I know how > to do that though. To do this I think we *need* to provide beta tester a validator kind of tool which they can use even without having an application unit test suite. And a way to easily report success or failure, and in case of success, a notion of their tests coverage (which is reduced to the list of PG features the application exercices, but an automated way to extract the information while running the tool would allow for hackers friendly categories). Anyone interrested into starting a project and coding the tool we so much need? Regards, -- dim PS: of course provided with such a tool, I'd run it for several applications as soon as possible, which might include every alpha release. PPS: I'll be happy to participate into such a project, but I seem to have a rather full plate already and a shrinking OpenSource free time, so waiting for me to get there won't help until it's about too late.
On Sun, 2009-08-23 at 01:57 -0400, Tom Lane wrote: > Robert Haas <robertmhaas@gmail.com> writes: > > I posted a note about a week ago which drew far less commentary than I > > expected, regarding the timetable for the release of 8.5. > > > http://archives.postgresql.org/pgsql-hackers/2009-08/msg01256.php > > > Perhaps this is already being discussed on -core, but if so the > > conclusions haven't been shared publicly. > > Core hasn't discussed 8.5 schedule since the discussions that Peter > summarized in the message you cited above. I share your concern that > "release in time for PGCon" isn't very realistic if we don't get more > aggressive about schedule. On the other hand, we didn't get all that > much done in this fest. If we cut back to a three-fest schedule > I think we may succeed in releasing 8.5 early, but only because there > is nothing interesting in it :-(. Dunno where the right balance is. General comment on thread: The level of detailed planning happening now is a change for the community and in general I think it's a good thing. In the past we've always said it will be shipped when it's ready, and now we seem to be caught by our own rules. There's no need to make hard decisions now. Let's keep some flexibility in our thinking. If the structures give us problems, lets change the structures. The idea is the plans help the developers, not hinder them or make it harder to include big features. In my view it is important that I have a holistic view of what I am doing and that means it is very difficult for others to help in a way that doesn't merely hinder me (or Fujii-san). Speed of coding is not the issue here, nor is the number of hands on a keyboard. I don't share the doubts or fears expressed that the two big patches will not make it into Postgres in this release. We now have more people experienced in these areas of code than at any other time in our history. We have almost-complete solutions from experienced developers. In particular, I have faith in Fujii-san. I would appreciate it if somebody could send out some messages of calm, while I/we work. The time for open review will come around soon enough. Faith and patience. Please. No need for Fawlty Towers re-runs. -- Simon Riggs www.2ndQuadrant.com
On Thu, Aug 27, 2009 at 08:48:43PM +0100, Simon Riggs wrote: > > On Sun, 2009-08-23 at 01:57 -0400, Tom Lane wrote: > > Robert Haas <robertmhaas@gmail.com> writes: > > > I posted a note about a week ago which drew far less commentary > > > than I expected, regarding the timetable for the release of 8.5. > > > > > http://archives.postgresql.org/pgsql-hackers/2009-08/msg01256.php > > > > > Perhaps this is already being discussed on -core, but if so the > > > conclusions haven't been shared publicly. > > > > Core hasn't discussed 8.5 schedule since the discussions that > > Peter summarized in the message you cited above. I share your > > concern that "release in time for PGCon" isn't very realistic if > > we don't get more aggressive about schedule. On the other hand, > > we didn't get all that much done in this fest. If we cut back to > > a three-fest schedule I think we may succeed in releasing 8.5 > > early, but only because there is nothing interesting in it :-(. > > Dunno where the right balance is. > > General comment on thread: > > The level of detailed planning happening now is a change for the > community and in general I think it's a good thing. In the past > we've always said it will be shipped when it's ready, and now we > seem to be caught by our own rules. There's no need to make hard > decisions now. Let's keep some flexibility in our thinking. If the > structures give us problems, lets change the structures. The idea is > the plans help the developers, not hinder them or make it harder to > include big features. > > In my view it is important that I have a holistic view of what I am > doing and that means it is very difficult for others to help in a > way that doesn't merely hinder me (or Fujii-san). Speed of coding is > not the issue here, nor is the number of hands on a keyboard. > > I don't share the doubts or fears expressed that the two big patches > will not make it into Postgres in this release. We now have more > people experienced in these areas of code than at any other time in > our history. We have almost-complete solutions from experienced > developers. In particular, I have faith in Fujii-san. > > I would appreciate it if somebody could send out some messages of > calm, while I/we work. The time for open review will come around > soon enough. With all due respect, the time for open review is now. You have already tried closed development several times, and it each time has been, more or less, a spectacular failure. Cheers, David. -- David Fetter <david@fetter.org> http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fetter@gmail.com Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate
Robert Haas escribió: > What I want to do is address the concern about too much of any given > year being consumed by beta and CommitFest. I'm not sure I know how > to do that though. How much time were we in beta? I thought most time was spent trying to get to beta in the first place. -- Alvaro Herrera http://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc.
On Thu, Aug 27, 2009 at 3:53 PM, David Fetter <david@fetter.org> wrote:
Unlike Robert and Heikki, I don't see you contributing to or assisting Simon's work. And, while I may be wrong, I doubt that you assisted in funding any of Simon's work on hot standby either. As such, it's my opinion that continuing to criticize him from the sidelines is not only rude, but is also a bad idea as it relates to his motivation in working on this feature.
-- With all due respect, the time for open review is now. You have> I would appreciate it if somebody could send out some messages of
> calm, while I/we work. The time for open review will come around
> soon enough.
already tried closed development several times, and it each time has
been, more or less, a spectacular failure.
Unlike Robert and Heikki, I don't see you contributing to or assisting Simon's work. And, while I may be wrong, I doubt that you assisted in funding any of Simon's work on hot standby either. As such, it's my opinion that continuing to criticize him from the sidelines is not only rude, but is also a bad idea as it relates to his motivation in working on this feature.
Jonah H. Harris, Senior DBA
myYearbook.com
On Thu, Aug 27, 2009 at 3:56 PM, Alvaro Herrera<alvherre@commandprompt.com> wrote: > Robert Haas escribió: >> What I want to do is address the concern about too much of any given >> year being consumed by beta and CommitFest. I'm not sure I know how >> to do that though. > > How much time were we in beta? I thought most time was spent trying to > get to beta in the first place. [ looks ] The final CommitFest began November 11, 2008. It closed March 25, 2009 (+ 144 days). Beta1 was released April 15, 2009 (+ 21 days). 8.4.0 was released July 1, 2009 (+ 77 days). The first CommitFest for 8.5 began on July 15, 2009 (+ 14 days). http://www.postgresql.org/about/news.1074 http://wiki.postgresql.org/index.php?title=CommitFest_2008-11&action=history http://www.postgresql.org/docs/8.4/static/release-8-4.html In total, the tree was closed for 256 days, or 8.5 months, of which the final CommitFest accounted for approximately 56%. Had we closed the final CommitFest in 30 days rather than 144 days, and had everything else taken the same amount of time, the release would have occurred on March 9th and the first CommitFest for 8.5 would have started on March 23rd. Hmm... maybe that's not actually that bad. If we stuck to a similar schedule for 8.5, but with a timely last CF, then we'd have either (3 CF): 2009-11-15 Final CommitFest Begins 2009-12-15 Final CommitFest Ends 2010-01-05 Beta 2010-03-23 Release 2010-04-06 First CommitFest for 8.6 Begins Or (4 CF): 2010-01-15 Final CommitFest Begins 2010-02-15 Final CommitFest Ends 2010-03-08 Beta 2010-05-24 Release 2010-06-07 First CommitFest for 8.6 Begins Of course I don't think we'd actually need to start a CommitFest quite as quickly as we did this time, because with a shorter release cycle there ought to be a lot less patches already accumulated by the time we release, especially if there are clearly defined tasks for developers to do during the beta period. On the other hand, 8.4beta was arguably too short, since we missed some serious problems, so the picture above may be a bit too rosy. ...Robert
andrew@dunslane.net (Andrew Dunstan) writes: > Actually, what I had in mind was getting people to run their > applications etc. in some non-production environment on the beta. I > talked to a client today and he said "sure, we have several > development environments and we can put one or two on the beta and > then let the developers just do their thing on it." Testing the things > we know about is in a way less important than making sure nothing else > got broken. I've gotten the DB work on one of our applications to the point where there's a meaningful set of DB tests that can be run in automated fashion. As a result, I rotate between PG versions periodically; every couple weeks, I recompile HEAD, and run a test against it to make sure I don't see any regressions. It would be insane to think about deploying on some 8.5-alpha version, but it's nice to have something I can run in ~5 minutes that exercises a fair bit of at least vaguely realistic functionality. -- "cbbrowne","@","ca.afilias.info" Christopher Browne "Bother," said Pooh, "Eeyore, ready two photon torpedoes and lock phasers on the Heffalump, Piglet, meet me in transporter room three"
On Thu, Aug 27, 2009 at 04:22:58PM -0400, Jonah H. Harris wrote: > On Thu, Aug 27, 2009 at 3:53 PM, David Fetter <david@fetter.org> wrote: > > > > I would appreciate it if somebody could send out some messages > > > of calm, while I/we work. The time for open review will come > > > around soon enough. > > > > With all due respect, the time for open review is now. You have > > already tried closed development several times, and it each time > > has been, more or less, a spectacular failure. > > Unlike Robert and Heikki, I don't see you contributing to or > assisting Simon's work. My assistance is of the form of actually getting it done. Simon's work is absolutely fantastic, but only when the rest of the community can actually help. When it shows up late, it actually hurts everybody, most of all Simon. > And, while I may be wrong, I doubt that you assisted in funding any > of Simon's work on hot standby either. As such, it's my opinion > that continuing to criticize him from the sidelines is not only > rude, but is also a bad idea as it relates to his motivation in > working on this feature. If, "your past strategy has a track record of failure, go with a new strategy, one pretty much universally adopted in PostgreSQL," will demotivate someone, I can't help that, and I doubt it's actually true. I'm trying to help here, and encouraging a failed strategy is not helping. Cheers, David. -- David Fetter <david@fetter.org> http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fetter@gmail.com Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate
Robert Haas <robertmhaas@gmail.com> wrote: > The final CommitFest began November 11, 2008. It closed March 25, > 2009 (+ 144 days). Beta1 was released April 15, 2009 (+ 21 days). I'm not entirely clear on what was happening during the 21 days between the end of the CommitFest and and the release of beta1. I seem to remember Bruce saying that there were bugs being fixed, and that it didn't make sense to release a beta with known bugs of that magnitude, but I'm not clear on what was up with that. Did we close the CF with known bugs open, or were these missed in the CF and found after? Surely it shouldn't normally take three weeks to get a beta test version to the public after we close the last CF? Just looking for where we could pick up a few weeks more of development in each year.... -Kevin
Robert Haas escribió: > Of course I don't think we'd actually need to start a CommitFest quite > as quickly as we did this time, because with a shorter release cycle > there ought to be a lot less patches already accumulated by the time > we release, especially if there are clearly defined tasks for > developers to do during the beta period. On the other hand, 8.4beta > was arguably too short, since we missed some serious problems, so the > picture above may be a bit too rosy. Maybe what this says is that we need to get a "pre-beta" release out as early as possible, just after finalizing the last commitfest, and before the "open items" and Bruce-approved release note writing are sorted out. Probably the last alpha release will fill that role. Sadly, the greek folk did not consider having a letter between alpha and beta :-( -- Alvaro Herrera http://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc.
Kevin Grittner escribió: > Robert Haas <robertmhaas@gmail.com> wrote: > > > The final CommitFest began November 11, 2008. It closed March 25, > > 2009 (+ 144 days). Beta1 was released April 15, 2009 (+ 21 days). > > I'm not entirely clear on what was happening during the 21 days > between the end of the CommitFest and and the release of beta1. I > seem to remember Bruce saying that there were bugs being fixed, and > that it didn't make sense to release a beta with known bugs of that > magnitude, but I'm not clear on what was up with that. That's nonsense IMHO. We should have just released a version documenting the known bugs, and asking for people to test the rest of the system. -- Alvaro Herrera http://www.CommandPrompt.com/ PostgreSQL Replication, Consulting, Custom Development, 24x7 support
"Kevin Grittner" <Kevin.Grittner@wicourts.gov> writes:
> Robert Haas <robertmhaas@gmail.com> wrote:
>> The final CommitFest began November 11, 2008. It closed March 25,
>> 2009 (+ 144 days). Beta1 was released April 15, 2009 (+ 21 days).
> I'm not entirely clear on what was happening during the 21 days
> between the end of the CommitFest and and the release of beta1.
Release note drafting is one part of it, but mostly it's "loose end
cleanup". Historically there have always been a pile of loose ends
to be dealt with, and the CommitFest structure doesn't really do
anything to avoid that. If you're interested, attached are all the
commits between commitfest closure (which I announced immediately
after committing the addition of contrib/btree_gin) and stamping
beta1 (which was actually several days before the date Robert gives,
because of the need for the packagers to do their thing).
It appears to me that release notes weren't actually the bottleneck this
time, though they have been in some prior cycles. Bruce committed a
first draft immediately after the commitfest closed. Rather, it was
arguing about things like \df behavior and cardinality() that took up
the time.
We could certainly have released a perhaps-less-polished beta earlier.
I think that the traditional criterion is that we don't release beta1
as long as there are any known issues that might force an initdb.
We were successful in avoiding a post-beta initdb this time, although
IIRC the majority of release cycles have had one --- so maybe you
could argue that that's not so important. It would certainly be
less important if we had working pg_migrator functionality to ease
the pain of going from beta to final.
Now that we have the alpha-release mechanism defined, some of the
pressure for a quick beta could be taken off by releasing a final
alpha right after the final commitfest closes.
regards, tom lane
2009-04-09 20:20 scrappy
* configure, configure.in, doc/bug.template,src/include/pg_config.h.win32: commit and tag beta1
2009-04-09 19:22 tgl
* doc/src/sgml/release.sgml: Update release notes throughyesterday; some minor wordsmithing.
2009-04-09 18:32 momjian
* doc/src/sgml/monitoring.sgml: Clarify documentation references topg_stat_get_blocks_fetched and
pg_stat_get_blocks_hit,persuggestion from Robert Haas.
2009-04-09 17:50 momjian
* src/tools/RELEASE_CHANGES: No more need to update FAQs.
2009-04-09 17:35 petere
* src/tools/RELEASE_CHANGES: Add URL for config.guess/sub updates
2009-04-09 17:33 petere
* config/: config.guess, config.sub: Update config.guess andconfig.sub
2009-04-09 16:50 tgl
* src/timezone/data/: africa, asia, leapseconds, northamerica,southamerica (REL8_1_STABLE), africa, asia,
leapseconds,northamerica,southamerica (REL8_3_STABLE), africa, asia,leapseconds, northamerica, southamerica
(REL8_0_STABLE),africa,asia, leapseconds, northamerica, southamerica (REL8_2_STABLE),africa, asia, leapseconds,
northamerica,southamerica: Update timezone data files to tzdata release 2009e: DST law changes inArgentina/San_Luis,
Cuba,Jordan (historical correction only),Morocco, Palestine, Syria, Tunisia.
2009-04-09 15:38 petere
* src/: backend/nls.mk, backend/po/de.po, backend/po/es.po,backend/po/fr.po, backend/po/ja.po,
backend/po/nl.po,backend/po/ru.po,backend/po/tr.po, bin/initdb/nls.mk,bin/initdb/po/de.po, bin/initdb/po/es.po,
bin/initdb/po/fr.po,bin/initdb/po/ja.po,bin/initdb/po/ru.po, bin/pg_config/nls.mk,bin/pg_config/po/de.po,
bin/pg_config/po/fr.po,bin/pg_config/po/ja.po,bin/pg_config/po/ru.po,bin/pg_config/po/tr.po,
bin/pg_controldata/nls.mk,bin/pg_controldata/po/de.po,bin/pg_controldata/po/fr.po,bin/pg_controldata/po/ja.po,
bin/pg_ctl/nls.mk,bin/pg_ctl/po/cs.po,bin/pg_ctl/po/de.po, bin/pg_ctl/po/fr.po,bin/pg_ctl/po/ja.po,
bin/pg_ctl/po/ru.po,bin/pg_dump/nls.mk,bin/pg_dump/po/de.po, bin/pg_dump/po/fr.po,
bin/pg_dump/po/ja.po,bin/pg_dump/po/tr.po,bin/pg_resetxlog/nls.mk,bin/pg_resetxlog/po/de.po,
bin/pg_resetxlog/po/fr.po,bin/pg_resetxlog/po/ja.po,bin/pg_resetxlog/po/ru.po,bin/psql/nls.mk, bin/psql/po/de.po,
bin/psql/po/es.po,bin/psql/po/fr.po,bin/psql/po/ja.po, bin/psql/po/tr.po,bin/scripts/nls.mk, bin/scripts/po/de.po,
bin/scripts/po/fr.po,bin/scripts/po/ja.po,interfaces/ecpg/ecpglib/nls.mk,interfaces/ecpg/ecpglib/po/de.po,
interfaces/ecpg/ecpglib/po/es.po,interfaces/ecpg/ecpglib/po/fr.po,
interfaces/ecpg/preproc/nls.mk,interfaces/ecpg/preproc/po/de.po,
interfaces/ecpg/preproc/po/es.po,interfaces/ecpg/preproc/po/fr.po,interfaces/libpq/nls.mk,interfaces/libpq/po/de.po,
interfaces/libpq/po/fr.po,interfaces/libpq/po/ja.po,pl/plperl/nls.mk, pl/plperl/po/de.po,pl/plperl/po/es.po,
pl/plperl/po/fr.po,pl/plpgsql/src/nls.mk,pl/plpgsql/src/po/de.po, pl/plpgsql/src/po/es.po,pl/plpgsql/src/po/fr.po,
pl/plpgsql/src/po/ja.po,pl/plpgsql/src/po/ro.po,pl/plpython/nls.mk, pl/plpython/po/de.po,pl/plpython/po/es.po,
pl/plpython/po/fr.po,pl/tcl/nls.mk,pl/tcl/po/de.po, pl/tcl/po/es.po, pl/tcl/po/fr.po: Translationupdates for 8.4 beta
2009-04-09 15:07 tgl
* doc/src/sgml/gin.sgml: Update GIN limitations documentation tomatch current reality.
2009-04-09 13:39 tgl
* doc/src/sgml/func.sgml,
doc/src/sgml/release.sgml,src/backend/utils/adt/arrayfuncs.c,src/include/catalog/catversion.h,
src/include/catalog/pg_proc.h,src/include/utils/array.h,
src/test/regress/expected/arrays.out,src/test/regress/sql/arrays.sql:Remove SQL-compatibility functioncardinality().
Itis not exactly clear how this ought to behavefor multi-dimensional arrays. Per discussion, not having it at allseems
betterthan having it with what might prove to be the wrongbehavior. We can always add it later when we have consensus
onthecorrect behavior.
2009-04-09 12:20 momjian
* doc/src/sgml/wal.sgml: Improve documentation about how checkpointspreads I/O activity.
2009-04-09 10:21 tgl
* src/backend/utils/misc/guc-file.l: Treat EOF like \n forline-counting purposes in ParseConfigFile, per bug #4752.
FujiiMasao
2009-04-08 22:57 tgl
* src/: pl/plpgsql/src/pl_exec.c, pl/plpgsql/src/plpgsql.h,test/regress/expected/plpgsql.out,
test/regress/sql/plpgsql.sql:Fixthe plpgsql memory leak exhibited in bug #4677. That leak wasintroduced by my patch of
2007-01-28to use per-subtransactionExprContexts/EStates: since we re-prepared any expression tree whenthe current
subtransactionID changed, we'd accumulate more andmore leaked expression state trees in the outermost subtransactionif
thesame function was executed at multiple levels ofsubtransaction nesting. To fix, go back to the previous schemewhere
therewas only one EState per transaction for simple plpgsqlexpressions. We really only need an ExprContext
persubtransaction,not a whole EState, so it's possible to keepprepared expression state trees in the one EState
throughoutthetransaction. This should be more efficient as well as not leakingmemory for cases involving lots of
subtransactions.Theadded regression test is the case that inspired the 2007-01-28patch in the first place, just to make
surewe didn't go backwards. The current memory leak complaint is unfortunately hard to testfor in the regression test
framework,though manual testing showsit's fixed.Although this is a pre-existing bug, I'm not back-patching becauseI'd
liketo see this method get some field testing first. Considerback-patching if it gets through 8.4beta unscathed.
2009-04-08 18:29 tgl
* doc/src/sgml/ref/psql-ref.sgml, src/bin/psql/describe.c: Removepsql's ancient hack that suppressed functions taking
orreturningcstring from the output of \df. Now that the default behavior isto exclude all system functions, the
de-clutteringrationale forthis behavior seems pretty weak; and it was always quiteconfusing/unhelpful if you were
actuallylooking for I/O functions. (Not to mention if you were looking for encoding converters orother cases that might
takeor return cstring.)
2009-04-08 18:08 tgl
* src/: backend/utils/adt/numeric.c,test/regress/expected/numeric.out: Allow leading and trailingspaces around NaN in
numeric_in.SamMason, rewritten a bit by Tom.
2009-04-08 17:51 petere
* src/: backend/executor/execQual.c, backend/utils/adt/xml.c,include/nodes/execnodes.h,
test/regress/expected/xml.out,test/regress/expected/xml_1.out,test/regress/sql/xml.sql:XMLATTRIBUTES() should send the
attributevalues throughmap_sql_value_to_xml_value() instead of directly through the datatype output function. This is
perSQL standard, and consistentwith XMLELEMENT().
2009-04-08 15:02 heikki
* src/bin/pg_dump/: pg_dump.c, pg_dumpall.c: Quote string literalscorrectly in the new CREATE SERVER statements and
binaryupgradeUPDATE statements.
2009-04-08 09:08 heikki
* src/backend/utils/init/postinit.c: Oops, mustn't calltextdomain() when compiling without --enable-nls
2009-04-08 05:50 heikki
* src/: backend/utils/init/miscinit.c,backend/utils/init/postinit.c, backend/utils/mb/mbutils.c,include/mb/pg_wchar.h:
Tellgettext which codeset to use by callingbind_textdomain_codeset(). We already did that on Windows, but it'sneeded on
otherplatforms too when LC_CTYPE=C. With other locales,we enforce (or trust) that the codeset of the locale matches
theserverencoding so we don't need to bind it explicitly. It shoulddo no harm in that case either, but I don't have
fullfaith in thePG encoding -> OS codeset mapping table yet. Per recent discussionon pgsql-hackers.
2009-04-08 00:05 momjian
* src/bin/psql/tab-complete.c: Improve tab completion for \ef.Andrew Gierth
2009-04-07 19:27 momjian
* src/backend/utils/misc/guc.c: Revert addition of units to GUCdescriptions; doesn't affect postgresql.conf.
2009-04-07 18:48 momjian
* configure, configure.in: Disable effective_io_concurrency onSolaris because posix_fadvise() is no-op on that
platform.
2009-04-07 18:22 momjian
* src/backend/utils/misc/: guc.c, postgresql.conf.sample: More GUCunits doc updates.Euler Taveira de Oliveira
2009-04-07 17:30 momjian
* doc/src/sgml/release.sgml: Add attribution for: Add Japanese message translations (Japan PostgreSQL
UsersGroup)
2009-04-07 17:29 momjian
* doc/src/sgml/release.sgml: Add release note item: Add Japanese message translations
2009-04-07 17:20 momjian
* doc/: FAQ, FAQ_DEV: Remove FAQ and FAQ_DEV ASCII and HTML filesfrom CVS; now on the wiki.Per-language files kept for
transatorusage.
2009-04-07 15:35 mha
* src/tools/msvc/Mkvcbuild.pm: Support Perl 5.10 and TCL 8.5 inMSVC builds.We should probably have a better way to do
this(meaning somethingnot hardcoded) eventually, but this fixes the problem for 8.4.Dave Page
2009-04-07 14:10 tgl
* contrib/pg_freespacemap/: pg_freespacemap.c (REL8_3_STABLE),pg_freespacemap.c (REL8_2_STABLE): Fix
contrib/pg_freespacemap'sunderestimateof the number of pages it could find in the FSM. Perreport from Dimitri
Fontaineand Andrew Gierth.(Affects only 8.2 and 8.3 since HEAD no longer has MaxFSMPages atall.)
2009-04-07 13:57 tgl
* contrib/pg_freespacemap/pg_freespacemap.c: Remove useless(leftover?) extern declaration.
2009-04-07 13:49 teodor
* src/backend/access/gist/gist.c (REL7_4_STABLE): Fix 'all at onepage bug' in picksplit method of R-tree emulation. Add
defensefrombuggy user-defined picksplit to GiST.
2009-04-07 13:46 teodor
* src/backend/access/gist/: gistproc.c, gistutil.c (REL8_1_STABLE),gist.c (REL8_0_STABLE): Fix 'all at one page bug' in
picksplitmethodof R-tree emulation. Add defense from buggy user-definedpicksplit to GiST.
2009-04-07 11:53 tgl
* contrib/fuzzystrmatch/: fuzzystrmatch.h (REL7_4_STABLE),fuzzystrmatch.h (REL8_1_STABLE), fuzzystrmatch.h
(REL8_3_STABLE),fuzzystrmatch.h(REL8_0_STABLE), fuzzystrmatch.h (REL8_2_STABLE),fuzzystrmatch.c: Defend against
non-ASCIIletters in fuzzystrmatchcode. The functions still don't behave very sanely for multibyteencodings, but at
leastthey won't be indexing off the ends ofstatic arrays.
2009-04-07 00:02 momjian
* doc/src/sgml/trigger.sgml: Add doc link to section about how tocompile triggers.
2009-04-06 20:31 tgl
* doc/src/sgml/backup.sgml,
doc/src/sgml/func.sgml,src/backend/access/transam/xlog.c,src/backend/catalog/system_views.sql,src/include/catalog/catversion.h,
src/include/catalog/pg_proc.h:Addan optional parameter to pg_start_backup() that specifieswhether to do the checkpoint
inimmediate or lazy mode. This is toaddress complaints that pg_start_backup() takes a long time evenwhen there's no
needto minimize its I/O consumption.
2009-04-06 17:00 momjian
* src/backend/utils/misc/: guc.c, postgresql.conf.sample: Add unitdocumentation for various postgresql.conf settings.
2009-04-06 15:34 petere
* src/backend/utils/mb/mbutils.c: Add entry in the encoding numberto OS name table for KOI8-U.
2009-04-06 15:03 momjian
* src/backend/utils/misc/postgresql.conf.sample: Properly alignequals signs in new postgresql.conf units comments.
2009-04-06 15:00 momjian
* src/backend/utils/misc/postgresql.conf.sample: Document inpostgresql.conf that the default units
forlog_min_duration_statementis milliseconds.
2009-04-06 14:40 momjian
* src/backend/utils/misc/postgresql.conf.sample: Displaypostgresql.conf unit options in an easier-to-understand,
2-columnformat.
2009-04-06 13:56 momjian
* doc/src/sgml/maintenance.sgml: Doc change in new patch,stand-alone -> standalone
2009-04-06 13:55 momjian
* doc/src/sgml/maintenance.sgml: Add documentation mention of'check_postgres.pl' in Routine Database Maintenance Tasks
section.
2009-04-06 11:50 momjian
* src/bin/psql/tab-complete.c: Adjust psql tab completion for new\d 'S' flag behavior; adjust code to be more flexible
aboutadditionalmodifiers for \d commands.
2009-04-06 11:43 tgl
* doc/src/sgml/fuzzystrmatch.sgml: Document the fact thatfuzzystrmatch doesn't work in multibyte encodings.
2009-04-06 11:01 tgl
* doc/src/sgml/keywords.sgml: Correct keywords table for status ofCOLLATE vs LC_COLLATE.
2009-04-06 10:47 teodor
* src/backend/access/gist/: gistproc.c, gistsplit.c(REL8_2_STABLE): Fix 'all at one page bug' in picksplit method
ofR-treeemulation. Add defense from buggy user-defined picksplit toGiST.
2009-04-06 10:39 teodor
* src/backend/access/gist/: gistproc.c, gistsplit.c(REL8_3_STABLE): Fix 'all at one page bug' in picksplit method
ofR-treeemulation. Add defense from buggy user-defined picksplit toGiST.
2009-04-06 10:27 teodor
* src/backend/access/gist/: gistproc.c, gistsplit.c: Fix 'all atone page bug' in picksplit method of R-tree emulation.
Adddefensefrom buggy user-defined picksplit to GiST.
2009-04-06 04:42 heikki
* doc/src/sgml/charset.sgml,
doc/src/sgml/keywords.sgml,doc/src/sgml/ref/create_database.sgml,src/backend/commands/dbcommands.c,
src/backend/parser/gram.y,src/bin/pg_dump/pg_dump.c,src/bin/pg_dump/pg_dumpall.c,src/bin/scripts/createdb.c,
src/include/parser/kwlist.h,src/interfaces/ecpg/preproc/ecpg.trailer:Rename the new CREATEDATABASE options to set
collationand ctype into LC_COLLATE andLC_CTYPE, per discussion on pgsql-hackers.
2009-04-05 18:28 tgl
* src/: backend/utils/adt/arrayfuncs.c,include/catalog/catversion.h, include/catalog/pg_proc.h,include/utils/array.h:
Changecardinality() into a C-code function,instead of a SQL-language alias for array_length(v,1). Theefficiency gain
hereis doubtless negligible --- what I'minterested in is making sure that if we have second thoughts aboutthe
definition,we will not have to force a post-beta initdb tochange the implementation.
2009-04-05 16:32 tgl
* src/backend/executor/execQual.c: Make ExecInitExpr build the listof SubPlans found in a plan tree in order of
discovery,rather thanreverse order. This doesn't matter functionally (I suppose theprevious coding dates from the
timewhen lcons was markedly cheaperthan lappend). However now that EXPLAIN is labeling subplans withIDs that are
basedon order of creation, this may help produce aslightly less surprising printout.
2009-04-05 15:59 tgl
* src/: backend/commands/explain.c, backend/nodes/copyfuncs.c,backend/nodes/equalfuncs.c,
backend/nodes/outfuncs.c,backend/optimizer/plan/subselect.c,backend/utils/adt/ruleutils.c,include/nodes/primnodes.h:
ChangeEXPLAIN output so that subplansand initplans (particularly CTEs) are individually labeled, ratherthan just
groupedunder an "InitPlan" or "SubPlan" heading. Thisin turn makes it possible for decompilation of a subplan
referencetousefully identify which subplan it's referencing. I also madeInitPlans identify which parameter symbol(s)
theycompute, so thatreferences to those parameters elsewhere in the plan tree can beconnected to the initplan that will
beexecuted. Per a gripe fromRobert Haas about EXPLAIN output of a WITH query being inadequate,plus some longstanding
petpeeves of my own.
2009-04-05 07:32 teodor
* src/backend/access/gin/: ginget.c, ginscan.c: Fix infinite loopwhile checking of partial match in pending list.
Improvecomments.Now GIN-indexable operators should be strict. Per Tom'squestions/suggestions.
2009-04-05 00:19 tgl
* src/: backend/postmaster/postmaster.c, bin/initdb/initdb.c,bin/pg_ctl/pg_ctl.c,
bin/pg_dump/pg_dump.c,bin/pg_dump/pg_dumpall.c,bin/pg_dump/pg_restore.c,bin/psql/startup.c, bin/scripts/common.c,
bin/scripts/common.h,include/getopt_long.h,port/getopt_long.c: Remove a boatload ofuseless definitions of 'int
optreset'. If we are using our ownports of getopt or getopt_long, those will define the variable forthemselves; and
ifnot, we don't need these, because we never touchthe variable anyway.
2009-04-05 00:09 tgl
* src/include/pg_config.h.win32: I had always wondered whypg_config.h.win32 claimed that Windows provides optreset.
Currentmastodonresults prove that in fact it does not; it was onlybecause getopt.c defined the variable anyway that
thingsfailed tofall over.
2009-04-04 20:40 tgl
* contrib/intarray/_int.sql.in,contrib/intarray/uninstall__int.sql, doc/src/sgml/intarray.sgml:Remove
contrib/intarray'sdefinitions of the <@ and @> operators,so that they don't cause confusion with the built-in
anyarrayversionsof those operators. Adjust the module's index opclassesto support the built-in operators in place of
theprivate ones.The private implementations are still available under theirhistorical names @ and ~, so no
functionalityis lost. Some quicktesting suggests that they offer no real benefit over the coreoperators, however.Per
acomplaint from Rusty Conover.
2009-04-04 18:36 tgl
* src/port/getopt.c: Hmm, baiji thinks we need explicit 'extern'here.
2009-04-04 17:55 tgl
* configure, configure.in, src/include/pg_config.h.in,src/port/getopt.c: Make an attempt at fixing our current Solaris
11breakage:add a configure probe for opterr (exactly like the onefor optreset) and have getopt.c define the variables
onlyifconfigure doesn't find them in libc.
2009-04-04 17:12 tgl
* src/: backend/access/common/reloptions.c,backend/commands/define.c,
backend/commands/foreigncmds.c,backend/commands/sequence.c,backend/commands/typecmds.c,backend/commands/view.c,
backend/nodes/copyfuncs.c,backend/nodes/equalfuncs.c,backend/nodes/makefuncs.c,backend/nodes/outfuncs.c,
backend/parser/gram.y,backend/parser/parse_clause.c,include/commands/defrem.h,include/foreign/foreign.h,
include/nodes/makefuncs.h,include/nodes/nodes.h,include/nodes/parsenodes.h: Remove therecently added node types
ReloptElemand OptionDefElem in favor ofadding optional namespace and action fields to DefElem. Havingthree node types
thatdo essentially the same thing bloats the codeand leads to errors of confusion, such as in yesterday's bug
reportfromKhee Chin.
2009-04-04 13:40 tgl
* src/: backend/commands/indexcmds.c,backend/storage/ipc/procarray.c,
include/storage/lock.h,include/storage/procarray.h:A session that does not have any livesnapshots does not have to be
waitedfor when we are waiting forold snapshots to go away during a concurrent index build. Inparticular, this rule
letsus avoid waiting for idle-in-transactionsessions.This logic could be improved further if we had some way to wake
upwhenthe session we are currently waiting for goesidle-in-transaction. However that would be a significantly
morecomplex/invasivepatch, so it'll have to wait for some other day.Simon Riggs, with some improvements by Tom.
2009-04-04 00:53 tgl
* src/: backend/utils/adt/timestamp.c,test/regress/expected/interval.out, test/regress/sql/interval.sql(REL7_4_STABLE),
backend/utils/adt/timestamp.c,test/regress/expected/interval.out,test/regress/sql/interval.sql(REL8_1_STABLE),
backend/utils/adt/timestamp.c,test/regress/expected/interval.out,test/regress/sql/interval.sql(REL8_3_STABLE),
backend/utils/adt/timestamp.c,test/regress/expected/interval.out,test/regress/sql/interval.sql(REL8_0_STABLE),
backend/utils/adt/timestamp.c,test/regress/expected/interval.out,test/regress/sql/interval.sql(REL8_2_STABLE),
backend/utils/adt/timestamp.c,test/regress/expected/interval.out,test/regress/sql/interval.sql:Rewrite interval_hash()
sothat the hashcodes are equal for valuesthat interval_eq() considers equal. I'm not sure how thatfundamental
requirementescaped us through multiple revisions ofthis hash function, but there it is; it's been wrong
sinceinterval_hashwas first written for PG 7.1. Per bug #4748 fromRoman Kononov.Backpatch to all supported
releases.Thispatch changes the contents of hash indexes for intervalcolumns. That's no particular problem for PG 8.4,
sincewe'vebroken on-disk compatibility of hash indexes already; but it willrequire a migration warning note in the next
minorreleases of allexisting branches: "if you have any hash indexes on columns of typeinterval, REINDEX them after
updating".
2009-04-03 20:45 alvherre
* src/: backend/access/common/reloptions.c,include/access/reloptions.h: Disallow setting fillfactor for TOASTtables.To
implementthis without almost duplicating the reloption table,treat relopt_kind as a bitmask instead of an integer
value. Thisdecreases the range of allowed values, but it's not clear thatthere's need for that much values anyway.This
patchalso makes heap_reloptions explicitly a no-op forrelation kinds other than heap and TOAST tables.Patch by ITAGAKI
Takahirowith minor edits from me. (In particularI removed the bit about adding relation kind to an error message,which
Iintend to commit separately.)
2009-04-03 20:44 tgl
* src/bin/psql/describe.c: Improve obsolete comment.
2009-04-03 20:41 tgl
* src/bin/psql/describe.c: Make \dt \di and friends more consistentabout the treatment of TOAST tables and indexes; to
wit,never showeither. (You can examine them with plain \d if you're really soinclined.)
2009-04-03 20:39 tgl
* doc/src/sgml/ref/psql-ref.sgml: Minor wordsmithing ondescriptions of some \d commands.
2009-04-03 19:38 tgl
* src/bin/psql/help.c: Add missing help output for \ef option. Andrew Gierth
2009-04-03 19:27 tgl
* src/port/: path.c (REL8_3_STABLE), path.c (REL8_2_STABLE),path.c: Use (unsigned char) cast in argument of
pg_tolower().Maybe it works on Windows without that, but we shouldn't put badexamples where people might copy them.
Also,reformat slightly toimprove the odds that pgindent won't go nuts on this.
2009-04-03 14:17 tgl
* src/backend/storage/buffer/bufmgr.c: Add a comment documentingthe question of whether PrefetchBuffer should try to
protectanalready-existing buffer from being evicted. This was left as anopen issue when the posix_fadvise patch was
committed. I'm notsure there's any evidence to justify more work in this area, but weshould have some record about it
inthe source code.
2009-04-03 12:59 tgl
* src/pl/plpython/: plpython.c, expected/plpython_function.out,expected/plpython_test.out,
sql/plpython_function.sql,sql/plpython_test.sql:Defend against possible crash if a plpythonfunction does not specify
namesfor its arguments. Also add aregression test, since someone apparently changed every singleplpython test case to
useonly named parameters; else we'd havenoticed this sooner.Euler Taveira de Oliveira, per a report from Alvaro
2009-04-03 07:52 mha
* src/port/: path.c (REL8_3_STABLE), path.c (REL8_2_STABLE),path.c: Make directory name comparisons on Win32 case
insensitive.Thismethod will not catch all different ways since the localehandling in NTFS doesn't provide an easy way
todo that, but itwill hopefully solve the most common cases causing startup problemswhen the backend is found in the
systemPATH.Attempts to fix bug #4694.
2009-04-02 18:44 momjian
* doc/src/sgml/config.sgml: Document that Solaris can't useeffective_io_concurrency because of an ineffective
posix_fadvise().
2009-04-02 18:39 tgl
* src/: backend/executor/execQual.c, backend/executor/execUtils.c,include/nodes/execnodes.h: Refactor ExecProject and
associatedroutinesso that fast-path code is used for simple Var targetlistentries all the time, even when there are
otherentries that arenot simple Vars. Also, ensure that we prefetch attributes (withslot_getsomeattrs) for all Vars in
thetargetlist, even thoseburied within expressions. In combination these changes seem tosignificantly reduce the
runtimefor cases where tlists are mostlybut not exclusively Vars. Per my proposal of yesterday.
2009-04-02 16:59 momjian
* src/backend/: access/transam/slru.c, executor/execScan.c,executor/nodeAgg.c, executor/nodeGroup.c,
executor/nodeHash.c,executor/nodeHashjoin.c,executor/nodeLimit.c,executor/nodeMaterial.c,
executor/nodeMergejoin.c,executor/nodeNestloop.c,executor/nodeSetOp.c, executor/nodeSort.c,executor/nodeSubplan.c,
executor/nodeUnique.c,utils/probes.d:Revert DTrace patch from Robert Lor
2009-04-02 16:16 tgl
* src/pl/plpgsql/src/pl_exec.c: Minor codebeautification/consolidation.
2009-04-02 15:57 momjian
* src/backend/utils/misc/guc.c: Give a better error message whentrying to change "effective_io_concurrency" on systems
withoutposix_fadvise().
2009-04-02 15:20 momjian
* doc/src/sgml/plpgsql.sgml, src/pl/plpgsql/src/pl_exec.c: HavePL/pgSQL FETCH set DIAGNOSTICS ROW_COUNT.Andrew Gierth
2009-04-02 15:14 momjian
* src/backend/: access/transam/slru.c, executor/execScan.c,executor/nodeAgg.c, executor/nodeGroup.c,
executor/nodeHash.c,executor/nodeHashjoin.c,executor/nodeLimit.c,executor/nodeMaterial.c,
executor/nodeMergejoin.c,executor/nodeNestloop.c,executor/nodeSetOp.c, executor/nodeSort.c,executor/nodeSubplan.c,
executor/nodeUnique.c,utils/probes.d: Addsupport for additional DTrace probes.Robert Lor
2009-04-02 14:06 teodor
* contrib/hstore/hstore_io.c (REL8_2_STABLE): Fix memory allocationfor output of hstore type. Per
"maosen.zhang"<maosen.zhang@alibaba-inc.com>report.
2009-04-02 13:57 teodor
* contrib/hstore/: hstore_io.c (REL8_3_STABLE), hstore_io.c: Fixmemory allocation for output of hstore type. Per
"maosen.zhang"<maosen.zhang@alibaba-inc.com>report.
2009-04-02 13:38 momjian
* src/bin/psql/describe.c: Do not show information_schema in \d*commands, unless 'S' or pattern is specified.Martin
Pihlak
2009-04-02 13:30 tgl
* src/: backend/utils/mb/mbutils.c, include/mb/pg_wchar.h: FixSetClientEncoding() to maintain a cache of previously
selectedencodingconversion functions. This allows transaction rollback torevert to a previous client_encoding
settingwithout doing freshcatalog lookups. I believe that this explains and fixes the recentreport of "failed to
commitclient_encoding" failures.This bug is present in 8.3.x, but it doesn't seem prudent toback-patch the fix, at
leastnot till it's had some time for fieldtesting in HEAD.In passing, remove SetDefaultClientEncoding(), which was
usednowhere.
2009-04-02 11:15 momjian
* doc/src/sgml/ref/psql-ref.sgml, src/bin/psql/describe.c: Changepsql \d* display so 'S' _or_ a pattern include system
objects.
2009-04-01 23:51 tgl
* src/backend/utils/misc/: guc.c (REL8_3_STABLE), guc.c: Fix GUC'sreports of assign_hook failure to always include the
parametervaluewe failed to assign, even in "can't happen" cases. Motivatedby wondering what's going on in a recent
troublereport where"failed to commit" did happen.
2009-04-01 21:16 tgl
* src/pl/plpgsql/src/: pl_exec.c (REL8_3_STABLE), pl_exec.c(REL8_2_STABLE), pl_exec.c: plpgsql's
exec_simple_cast_value()mistakenlysupposed that it could bypass casting effort wheneverthe input value was NULL.
Howeverthis prevents application ofnot-null domain constraints in the cases that use this function, asillustrated in
bug#4741. Since this function isn't meant for usein performance-critical paths anyway, this certainly seems
likeanothercase of "premature optimization is the root of all evil".Back-patch as far as 8.2; older versions made no
effortto enforcedomain constraints here anyway.
2009-04-01 14:54 tgl
* src/bin/pg_dump/: pg_dump.c (REL8_3_STABLE), pg_dump.c: Improvepg_dump's query for retrieving BLOB comments to be
moreefficientwhen there are many blobs and not so many comments. Tamas Vincze
2009-04-01 05:17 heikki
* src/backend/utils/adt/pg_locale.c: Update comment to reflect thatLC_COLLATE and LC_CTYPE are now per-database
settings.
2009-03-31 23:32 tgl
* doc/src/sgml/func.sgml: Index some array functions, per MarioSplivalo.
2009-03-31 23:23 tgl
* src/backend/libpq/: auth.c, pg_hba.conf.sample: Remove lastreferences to the crypt auth method, per Andreas
Scherbaum.
2009-03-31 18:54 tgl
* contrib/: pgstattuple/pgstattuple.c (REL7_4_STABLE),pgstattuple/pgstattuple.c
(REL8_1_STABLE),pageinspect/btreefuncs.c,pageinspect/rawpage.c,pgstattuple/pgstatindex.c,
pgstattuple/pgstattuple.c(REL8_3_STABLE),pgstattuple/pgstattuple.c (REL8_0_STABLE),pgstattuple/pgstatindex.c,
pgstattuple/pgstattuple.c(REL8_2_STABLE),pageinspect/btreefuncs.c, pageinspect/rawpage.c,pgstattuple/pgstatindex.c,
pgstattuple/pgstattuple.c:Fixcontrib/pgstattuple and contrib/pageinspect to prevent attempts toread temporary tables of
othersessions; that is unsafe because ofthe way our buffer management works. Per report from StuartBishop. This is
redundantwith the bufmgr.c checks in HEAD, butnot at all redundant in the back branches.
2009-03-31 18:23 momjian
* doc/src/sgml/release.sgml: Remove some "Other" sections in therelease notes by putting the items at the top of their
sections.
2009-03-31 18:12 tgl
* src/: backend/catalog/index.c, backend/catalog/namespace.c,backend/catalog/toasting.c,
backend/commands/analyze.c,backend/commands/cluster.c,backend/commands/copy.c,backend/commands/indexcmds.c,
backend/commands/tablecmds.c,backend/commands/vacuum.c,
backend/optimizer/prep/prepunion.c,backend/postmaster/autovacuum.c,
backend/storage/buffer/bufmgr.c,backend/utils/cache/relcache.c,include/utils/rel.h: Modify therelcache to record the
tempstatus of both local and nonlocal temprelations; this is no more expensive than before, now that we
havepg_class.relistemp. Insert tests into bufmgr.c to preventattempting to fetch pages from nonlocal temp relations.
Thisprovidesa low-level defense against bugs-of-omission allowing temppages to be loaded into shared buffers, as in
thecontrib/pgstattupleproblem reported by Stuart Bishop. While atit, tweak a bunch of places to use new relcache
tests(instead ofexpensive probes into pg_namespace) to detect local or nonlocaltemp tables.
2009-03-31 14:58 mha
* src/bin/initdb/: initdb.c (REL8_1_STABLE), initdb.c(REL8_3_STABLE), initdb.c (REL8_0_STABLE),
initdb.c(REL8_2_STABLE),initdb.c: Don't crash initdb when we fail to getthe current username. Give an error message
andexit instead, likewe do elsewhere...Per report from Wez Furlong and Robert Treat.
2009-03-31 13:59 tgl
* doc/src/sgml/catalogs.sgml,
src/backend/catalog/heap.c,src/backend/utils/cache/relcache.c,src/include/catalog/catversion.h,src/include/catalog/pg_attribute.h,
src/include/catalog/pg_class.h:Adda "relistemp" boolean column to pg_class, which is true fortemporary relations
(includinga temp table's indexes and toasttable/index), and false for normal relations. For ease ofchecking, this
commitjust adds the column and fills it correctly--- revising the relation access machinery to use it will
comeseparately.
2009-03-31 01:18 heikki
* src/backend/storage/ipc/: procarray.c (REL8_1_STABLE),procarray.c (REL8_3_STABLE), procarray.c
(REL8_2_STABLE),procarray.c:Fix a rare race condition when commit_siblings > 0 anda transaction commits at the same
instantas a new backend isspawned. Since CountActiveBackends() doesn't hold ProcArrayLock, itneeds to be prepared for
thecase that a pointer at the end of theproc array is still NULL even though numProcs says it should bevalid, since it
doesn'thold ProcArrayLock. Backpatch to 8.1. 8.0and earlier had this right, but it was broken in the split ofPGPROC
andsinval shared memory arrays.Per report and proposal by Marko Kreen.
2009-03-30 22:34 momjian
* doc/src/sgml/release.sgml: Update release note introductorydescription.
2009-03-30 21:41 tgl
* doc/src/sgml/libpq.sgml, src/interfaces/libpq/exports.txt,src/interfaces/libpq/fe-secure.c,
src/interfaces/libpq/libpq-fe.h:AddPQinitOpenSSL() function to support applications that uselibcrypto but not OpenSSL
(orperhaps vice versa, if that'spossible).Andrew Chernow, with minor editorialization by me.
2009-03-30 21:26 momjian
* doc/src/sgml/release.sgml: More new subsections in release notes.
2009-03-30 18:01 momjian
* doc/src/sgml/release.sgml: More release note changes, including alower level of subsections.
2009-03-30 16:32 momjian
* doc/src/sgml/release.sgml: More release note adjustments,reordering.
2009-03-30 15:59 momjian
* doc/src/sgml/release.sgml: More release note wordingimprovements; section order adjustments.
2009-03-30 14:34 momjian
* doc/src/sgml/release.sgml: Reorder release note sections.
2009-03-30 13:30 tgl
* src/backend/optimizer/plan/planner.c: Fix window function plangeneration to cope with volatile sort expressions.
(Notclear howuseful these really are, but failing is no good...) Per report fromDavid Fetter and Robert Treat.
2009-03-30 12:15 alvherre
* doc/src/sgml/: plpython.sgml (REL8_3_STABLE), plpython.sgml:Update URL to Python bug tracker. Backpatch to 8.3;
doesn'tseemworthy of further backpatch.
2009-03-30 00:08 tgl
* src/: backend/access/common/heaptuple.c,backend/executor/execTuples.c,
include/executor/tuptable.h,test/regress/expected/rangefuncs.out,test/regress/sql/rangefuncs.sql
(REL8_3_STABLE),backend/access/common/heaptuple.c,backend/executor/execTuples.c,include/executor/tuptable.h,
test/regress/expected/rangefuncs.out,test/regress/sql/rangefuncs.sql(REL8_2_STABLE),backend/access/common/heaptuple.c,
backend/executor/execTuples.c,include/executor/tuptable.h,
test/regress/expected/rangefuncs.out,test/regress/expected/with.out,
test/regress/sql/rangefuncs.sql,test/regress/sql/with.sql:Fix an oversight in the support forstoring/retrieving
"minimaltuples" in TupleTableSlots. We havefunctions for retrieving a minimal tuple from a slot after storinga regular
tuplein it, or vice versa; but these were implemented byconverting the internal storage from one format to the other.
Theproblemwith that is it invalidates any pass-by-reference Datumsthat were already fetched from the slot, since
they'llbe pointinginto the just-freed version of the tuple. The known problem casesinvolve fetching both a whole-row
variableand a pass-by-referencevalue from a slot that is fed from a tuplestore or tuplesortobject. The added
regressiontests illustrate some simple cases,but there may be other failure scenarios traceable to the same bug. Note
thatthe added tests probably only fail on unpatched code ifit's built with --enable-cassert; otherwise the bug leads
tofetchingfrom freed memory, which will not have been overwrittenwithout additional conditions.Fix by allowing a slot
tocontain both formats simultaneously;which turns out not to complicate the logic much at all, ifanything it seems less
contortedthan before.Back-patch to 8.2, where minimal tuples were introduced.
2009-03-29 15:13 momjian
* doc/src/sgml/release.sgml: More release note markup.
2009-03-28 23:58 momjian
* doc/src/sgml/release.sgml: More release note markup.
2009-03-28 23:01 momjian
* doc/src/sgml/release.sgml: Add SGML markup forcommands/literal/application/etc in release notes; still more workto
do.
2009-03-28 18:05 momjian
* doc/src/sgml/release.sgml: Consistent 8.4 release note naming forItagaki Takahiro
2009-03-28 14:48 momjian
* src/interfaces/libpq/fe-secure.c: Clarify variable naming:pq_initssllib -> pq_init_ssl_lib
2009-03-28 10:15 momjian
* doc/src/sgml/release.sgml: Update release notes to say citext ismulti-byte aware, per suggestion from patch author:
Add /contrib/citext as a case-insensitive, multibyte-capable text data type (David Wheeler)
2009-03-27 23:26 momjian
* doc/src/sgml/ref/: alter_role.sgml, set_role.sgml: Betterdocument that SET ROLE does not uset ALTER ROLE SET
settings;suggestedwording from Josh Berkus.
2009-03-27 21:36 momjian
* doc/src/sgml/libpq.sgml, src/interfaces/libpq/fe-secure.c: Betterdocument PQinitSSL(0) behavior in regards to
libcrypto.
2009-03-27 20:10 tgl
* doc/src/sgml/monitoring.sgml: Add documentation of the fact thatdtrace probes evaluate their parameters even when not
active.Explain how to prevent that with an ENABLED() check.
2009-03-27 18:39 momjian
* doc/src/sgml/release.sgml: Document in release notes that NOT INis only for NOT EXIST clauses.Andrew Gierth
2009-03-27 15:58 tgl
* configure, configure.in: On Solaris, we should only force use ofour own getopt(); it's okay to use the system's
getopt_long(). Theprevious coding was the result of a sloppy discussion that failedto draw this distinction. The
resultwas that PG programs don'thandle options as users of that platform expect. Per gripe fromChuck McDevitt.Although
thisis a pre-existing bug, I'm not backpatching since Ithink we could do with a bit of beta testing before concluding
thisisreally OK.
2009-03-27 15:17 mha
* doc/src/sgml/release.sgml: Fix markup, per Devrim
2009-03-27 14:56 tgl
* src/backend/utils/adt/xml.c: Add an errdetail explaining why wereject infinite dates and timestamps while converting
toXML. Bernd Helmle
2009-03-27 14:30 tgl
* src/: backend/executor/execQual.c,
backend/executor/functions.c,backend/executor/nodeCtescan.c,backend/executor/nodeFunctionscan.c,backend/executor/nodeMaterial.c,
backend/executor/nodeWindowAgg.c,backend/executor/nodeWorktablescan.c,
backend/tcop/pquery.c,backend/utils/sort/tuplestore.c,include/utils/tuplestore.h: Fixpossible failures when a
tuplestoreswitches from in-memory toon-disk mode while callers hold pointers to in-memory tuples. Ireported this for
thecase of nodeWindowAgg's primary scan tuple,but inspection of the code shows that all of the calls innodeWindowAgg
andnodeCtescan are at risk. For the moment, fix itwith a rather brute-force approach of copying whenever one of
theat-riskcallers requests a tuple. Later we might think of somesort of reference-count approach to reduce tuple
copying.
2009-03-27 11:57 tgl
* src/backend/catalog/index.c: Teach reindex_index() to clearpg_index.indcheckxmin when possible. Greg Stark, slightly
modifiedbyme.
2009-03-27 10:58 heikki
* src/bin/psql/: tab-complete.c (REL8_1_STABLE), tab-complete.c(REL8_3_STABLE), tab-complete.c (REL8_2_STABLE),
tab-complete.c:Fixtab completion of ANALYZE VERBOSE <tab>. It was previouslyconfused with EXPLAIN ANALYZE VERBOSE.Greg
SabinoMullane, reformatted by myself. Backpatch to 8.1, wherethe bug was introduced.
2009-03-27 08:01 mha
* doc/src/sgml/release.sgml: Clearify new SSL certificateverification in libpq
2009-03-27 07:58 mha
* doc/src/sgml/release.sgml: Fix release notes about pg_hba changes
2009-03-26 22:25 momjian
* doc/src/sgml/release.sgml: Updated release wording, per GregStark: Previously EXPLAIN VERBOSE had output an
internal representation of the
2009-03-26 21:44 momjian
* doc/src/sgml/release.sgml: Second batch of release note fixes byGuillaume Smet
2009-03-26 21:26 momjian
* doc/src/sgml/release.sgml: Mark Greg as the instigator of thestatistics target increase: Increase the default
valueof default_statistics_target from10 to 100 (Greg Sabino Mullane, Tom)
2009-03-26 20:45 momjian
* doc/src/sgml/release.sgml: First round of release note changesfrom Guillaume Smet.
2009-03-26 20:11 momjian
* doc/src/sgml/release.sgml: Fix release note attribution: Support the IS0 8601 time interval syntax (Tom, Kevin
Grittner)per Ron Mayer
2009-03-26 20:08 momjian
* doc/src/sgml/release.sgml: Improved release notes intervalwording: Made interval seconds rounding more
consistentacross output formats (Ron Mayer)Ron Mayer
2009-03-26 20:06 momjian
* doc/src/sgml/release.sgml: Remove duplicate interval (fractionalseconds) items; Ron Mayer
2009-03-26 20:01 momjian
* doc/src/sgml/release.sgml: Document that Datestyle no longercontrols interval output, per suggestion from Ron Mayer
2009-03-26 18:29 tgl
* contrib/pg_standby/: pg_standby.c (REL8_3_STABLE), pg_standby.c:Make pg_standby's maxretries option do what one would
expect.Fujii Masao
2009-03-26 18:26 petere
* src/: backend/catalog/dependency.c, backend/catalog/pg_proc.c,backend/catalog/pg_shdepend.c,
backend/executor/execQual.c,backend/parser/parse_func.c,backend/postmaster/bgwriter.c,bin/pg_dump/pg_backup_archiver.c,
bin/pg_dump/pg_backup_tar.c,bin/pg_dump/pg_dump.c,bin/psql/describe.c, bin/psql/print.c,include/c.h,
interfaces/ecpg/preproc/variable.c,pl/plpgsql/src/pl_exec.c,pl/plpython/plpython.c: Gettext pluralsupportIn the
backend,I changed only a handful of exemplary orimportant-looking instances to make use of the plural support;there is
probablymore work there. For the rest of the source,this should cover all relevant cases.
2009-03-26 16:55 tgl
* doc/src/sgml/: charset.sgml, config.sgml: Fix a couple of placesthat still claimed LC_COLLATE and LC_CTYPE can't be
changedafterinitdb.
2009-03-26 15:24 tgl
* src/backend/commands/copy.c: Adjust phrasing of complaints aboutmulti-byte COPY delimiter strings. Per pgsql-hackers
discussionof2009-02-17.
2009-03-26 13:15 tgl
* src/:
backend/nodes/outfuncs.c,backend/optimizer/path/costsize.c,backend/optimizer/plan/createplan.c,backend/optimizer/util/pathnode.c,
include/nodes/relation.h:If weexpect a hash join to be performed in multiple batches, suppress"physical tlist"
optimizationon the outer relation (ie, force aprojection step to occur in its scan). This avoids storing
uselesscolumnvalues when the outer relation's tuples are written totemporary batch files.Modified version of a patch by
MichaelHenderson and RamonLawrence.
2009-03-26 08:38 momjian
* doc/src/sgml/release.sgml: Correction: ansi-join ->anti-join.
2009-03-25 23:46 tgl
* doc/src/sgml/release.sgml: Fix markup so that 'make HISTORY'works. A couple very minor editorial improvements.
2009-03-25 22:48 momjian
* doc/src/sgml/release.sgml: Reorder 8.4 release note sections.
2009-03-25 22:40 momjian
* doc/src/sgml/release.sgml: Re-add release notes for release8.3.7.
2009-03-25 21:48 momjian
* doc/src/sgml/release.sgml: Adjust AS OF release notes markup.
2009-03-25 21:31 momjian
* doc/src/sgml/release.sgml: Mention release note items current asof 2009-03-16.
2009-03-25 21:19 momjian
* doc/src/sgml/release.sgml: First version of 8.4 release notes; markup/cleanup/reorganization still required.
On Thu, Aug 27, 2009 at 03:04:20PM -0400, Robert Haas wrote: > On Thu, Aug 27, 2009 at 2:54 PM, Peter Eisentraut<peter_e@gmx.net> wrote: > > On tor, 2009-08-27 at 09:58 -0400, Robert Haas wrote: > >> To get positive results in which you can have confidence, you > >> have to know that the testing which was done actually did a > >> reasonably good job exercising the code in a way that would have > >> flushed out bugs, had any been present. That sounds a lot like > >> the definition of a regression test suite. Of course, we have > >> that already, but it's nowhere near comprehensive. Maybe we > >> should be looking at an expanded test suite that runs on a time > >> scale of hours rather than seconds. Actually, didn't Peter talk > >> about something like this at PGCon? > > > > Let's look at it this way: If I were writing a compiler, then I > > would have two main test approaches. First, I would have an > > in-tree test suite that compiles a bunch of example code snippets > > and checks that the results are reasonable. Call that a > > regression test. It would be informed by code coverage analysis > > and previously reported bugs. Second, as part of my release > > cycle, I would have an agenda to try to compile a large set of > > real programs against my new compiler version. I would do that > > during the beta period. You will notice that GCC pretty much > > operates that way. > > > > We have regression tests. They could and should be expanded. > > That's a developer job, and we can start working on that now. > > But this discussion was about what to do during beta. And I > > think during beta you want to test PostgreSQL against a large set > > of real applications. But we could try to clarify how to actually > > do that in an organized way. > > > > Now, if you want to improve the regression tests, I would suggest > > going through the commits since 8.4beta and since 8.4.0 final > > release and ask how these problems could have been prevented or > > caught earlier. I suppose a test suite for WAL might be part of > > the answer, but a closer analysis might be insightful. > > What I want to do is address the concern about too much of any given > year being consumed by beta and CommitFest. I'm not sure I know how > to do that though. How about something in the alphas to the effect of, Using PostgreSQL? Have a development server to spare? Try your application stack on alpha1! We'd love to hear back. Functionality, performance, you name it. Cheers, David. -- David Fetter <david@fetter.org> http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fetter@gmail.com Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate
David Fetter wrote: > > How about something in the alphas to the effect of, > > Using PostgreSQL? > Have a development server to spare? > Try your application stack on alpha1! > We'd love to hear back. Functionality, performance, you name it. > > > I don't know of anyone who is likely to want to try out alphas in their normal development environments. The client I approached was specifically prepared to test beta releases that way. cheers andrew
Le jeudi 27 août 2009 à 14:27 -0500, Jaime Casanova a écrit : > the point was that if we simply were saying: hey! mysql can interpret > this, make postgres do the same then we could end up with a lot of > broken stuff... just because mysql users think is wonderful to not > have to write sane code... Not MySQL in general. Only the subset which helps and seems reasonable. Bye, JM
Simon, > The level of detailed planning happening now is a change for the > community and in general I think it's a good thing. In the past we've > always said it will be shipped when it's ready, and now we seem to be > caught by our own rules. There's no need to make hard decisions now. > Let's keep some flexibility in our thinking. If the structures give us > problems, lets change the structures. The idea is the plans help the > developers, not hinder them or make it harder to include big features. There's some very good reasons for the health of the project to have specific release dates and stick to them. "When will it be released" is an important question to answer, and it's far better for the developers who *aren't* working on big features to not be elastic. So, with that in mind: what is your statement on three versus four commitfests? Does it make a difference to you? -- Josh Berkus PostgreSQL Experts Inc. www.pgexperts.com
On Thu, Aug 27, 2009 at 6:09 PM, Tom Lane<tgl@sss.pgh.pa.us> wrote: > "Kevin Grittner" <Kevin.Grittner@wicourts.gov> writes: >> Robert Haas <robertmhaas@gmail.com> wrote: >>> The final CommitFest began November 11, 2008. It closed March 25, >>> 2009 (+ 144 days). Beta1 was released April 15, 2009 (+ 21 days). > >> I'm not entirely clear on what was happening during the 21 days >> between the end of the CommitFest and and the release of beta1. > > Release note drafting is one part of it, but mostly it's "loose end > cleanup". Historically there have always been a pile of loose ends > to be dealt with, and the CommitFest structure doesn't really do > anything to avoid that. If you're interested, attached are all the > commits between commitfest closure (which I announced immediately > after committing the addition of contrib/btree_gin) and stamping > beta1 (which was actually several days before the date Robert gives, > because of the need for the packagers to do their thing). > > It appears to me that release notes weren't actually the bottleneck this > time, though they have been in some prior cycles. Bruce committed a > first draft immediately after the commitfest closed. Rather, it was > arguing about things like \df behavior and cardinality() that took up > the time. It felt, at the time, like the release notes were holding things up, hence the various and so-far-unsuccesful volunteering related to that task. But it's possible that there was enough parallel activity going on that it wasn't actually so. > We could certainly have released a perhaps-less-polished beta earlier. > I think that the traditional criterion is that we don't release beta1 > as long as there are any known issues that might force an initdb. > We were successful in avoiding a post-beta initdb this time, although > IIRC the majority of release cycles have had one --- so maybe you > could argue that that's not so important. It would certainly be > less important if we had working pg_migrator functionality to ease > the pain of going from beta to final. > > Now that we have the alpha-release mechanism defined, some of the > pressure for a quick beta could be taken off by releasing a final > alpha right after the final commitfest closes. Definitely. Looking at it in hindsight, 3 weeks seems like a reasonable amount of time between the end of the last CommitFest and the start of beta. It felt long at the time, but maybe that's partly because the CommitFest was so intermininable. I think what a lot of this boils down to is that we need a better system for managing the tasks that need to be completed at each stage of the release process, and who is working on each one, and what the status of it is, just as we do for CommitFests. ...Robert
Andrew Dunstan wrote: > I don't know of anyone who is likely to want to try out alphas in their > normal development environments. The client I approached was > specifically prepared to test beta releases that way. Perhaps end-users won't, but I think companies who develop software that works on top of postgres will. Perhaps to make sure their existing software continues to work; or perhaps to get a head start working with new features. I test against CVS-head occasionally.
Josh Berkus wrote: > There's some very good reasons for the health of the project to have > specific release dates and stick to them. Help me understand why? The Linux kernel seems to do fine with a "when it is ready" cycle, where some releases(2.6.24) take twice the time of others(2.6.28)[1,2]. I imagine it has similar stability and lack-of-data-loss requirements as postgres does. I understand why commercial packagers like Ubuntu - especially public ones like Novell and Red Hat who have to forecast earnings - want to schedule their releases. And I can imagine they'd appreciate it if project releases aren't too close to their release schedules (if postgres releases right after they release they suffer from not having the current version; if postgres releases just before, they have limited testing time). [1] http://www.linuxfoundation.org/publications/linuxkerneldevelopment.php [2] http://fblinux.freebase.com/view/base/fblinux/views/linux_kernel_release > So, with that in mind: what is your statement on three versus four > commitfests? Does it make a difference to you?
On Fri, Aug 28, 2009 at 4:39 AM, Ron Mayer<rm_pg@cheapcomplexdevices.com> wrote:
> Josh Berkus wrote:
>> There's some very good reasons for the health of the project to have
>> specific release dates and stick to them.
>
> Help me understand why?
>
> The Linux kernel seems to do fine with a "when it is ready" cycle,
> where some releases(2.6.24) take twice the time of others(2.6.28)[1,2].
> I imagine it has similar stability and lack-of-data-loss requirements
> as postgres does.
Uhm, the Linux release process is actually now the canonical example
of time-based release management. They used to do "when it is ready"
and that led to such a dramatic catastrophic failure with 2.6 that
they adopted the most dogmatically time-based system of any open
source project.
They basically don't do any integration testing and leave that up to
the distributions now. Instead they have an "rc" release *every week*
like clockwork and every 2-3 months the last one becomes a new version
regardless of whether there's any notable new feature.
Other than the first few releases after 2.6.0 when things were still
fairly unstable and major fixes were going in the release cycle has
been remarkaby regular modulo holidays and vacations:
r | d | days
------------------+------------+------ChangeLog-2.6.30 | 2009-06-10 | 79ChangeLog-2.6.29 | 2009-03-23 |
89ChangeLog-2.6.28| 2008-12-24 | 75ChangeLog-2.6.27 | 2008-10-10 | 89ChangeLog-2.6.26 | 2008-07-13 |
87ChangeLog-2.6.25| 2008-04-17 | 84ChangeLog-2.6.24 | 2008-01-24 | 107ChangeLog-2.6.23 | 2007-10-09 |
92ChangeLog-2.6.22| 2007-07-09 | 74ChangeLog-2.6.21 | 2007-04-26 | 81ChangeLog-2.6.20 | 2007-02-04 |
67ChangeLog-2.6.19| 2006-11-29 | 70ChangeLog-2.6.18 | 2006-09-20 | 94ChangeLog-2.6.17 | 2006-06-18 |
90ChangeLog-2.6.16| 2006-03-20 | 76ChangeLog-2.6.15 | 2006-01-03 | 67ChangeLog-2.6.14 | 2005-10-28 |
60ChangeLog-2.6.13| 2005-08-29 | 73ChangeLog-2.6.12 | 2005-06-17 | 107ChangeLog-2.6.11 | 2005-03-02 |
68ChangeLog-2.6.10| 2004-12-24 | 66ChangeLog-2.6.9 | 2004-10-19 | 66ChangeLog-2.6.8 | 2004-08-14 |
59ChangeLog-2.6.7 | 2004-06-16 | 37ChangeLog-2.6.6 | 2004-05-10 | 36ChangeLog-2.6.5 | 2004-04-04 |
24ChangeLog-2.6.4 | 2004-03-11 | 22ChangeLog-2.6.3 | 2004-02-18 | 14ChangeLog-2.6.2 | 2004-02-04 |
26ChangeLog-2.6.1 | 2004-01-09 | 22ChangeLog-2.6.0 | 2003-12-18 |
(31 rows)
--
greg
http://mit.edu/~gsstark/resume.pdf
On Thu, 27 Aug 2009, Ron Mayer wrote: > The Linux kernel seems to do fine with a "when it is ready" cycle, > where some releases(2.6.24) take twice the time of others(2.6.28)[1,2]. > [2] http://fblinux.freebase.com/view/base/fblinux/views/linux_kernel_release That link has bad data. If you check the tagging dates by looking at the source material here: http://www.kernel.org/pub/linux/kernel/v2.6/ChangeLog-2.6.27 http://www.kernel.org/pub/linux/kernel/v2.6/ChangeLog-2.6.28 You can see that 2.6.28 didn't come out until 2008-12-14, not the 2008-10-24 claimed on the freebase.com site. > I imagine it has similar stability and lack-of-data-loss requirements > as postgres does. The Linux kernel developers are very clear that they don't care one bit about testing for stability or lack of data loss in any system-oriented way. That's somebody else's job now, typically the Linux distributor who decides which of the kernels floating around are the most stable, hopefully running more and larger tests than the kernel developers do. For example, if you consider Ubuntu 9.04 Jaunty, development started with 2.6.27, upgraded to 2.6.28, then rejected moving to 2.6.29 even though they might have slipped it in.[1] When faced with the similar decision for 2.6.26 vs. 2.6.27 in the previous release, they went for the later one, based on the features they needed to be stable.[2] It's really amazing that a useful result ever comes out of this model at all, and I know I'm not alone that I presume all Linux kernel releases are too full of bugs to be useful until I've proven otherwise with my own QA. If the core PostgreSQL development worked like the Linux kernel, at the end of each CommitFest whatever was done at that point would get published as the new release. Instead of pausing to focus on a stable release everyone would just speed ahead, backporting any major issues not discovered until after the official release. [1] http://www.h-online.com/news/No-2-6-29-kernel-for-Jaunty-Jackalope--/112636 [2] https://lists.ubuntu.com/archives/ubuntu-devel/2008-August/026142.html -- * Greg Smith gsmith@gregsmith.com http://www.gregsmith.com Baltimore, MD
On Thu, Aug 27, 2009 at 09:38:15PM +0200, Dimitri Fontaine wrote: > Exactly, and I think that what we're missing here is a simple tool for > our users to check a new PostgreSQL release against their existing > application. > > We already know how to either log all queries and analyze the log files > (CSV makes it easier, pgfouine parses them too) or to have a fe/be > protocol proxy to record application SQL traffic (tsung recorder does > that). > > What we miss is a tool to run the captured queries through both versions > of PG and report any resultset mismatch, of course with a way to account > for ordering issues (but we've seen people rely on the ordering when > they don't give an order by clause, then bug the lists about it if a new > release changes it). This would be very useful. I often am asked "how much better will the new release run our apps" as part of convincing a client to upgrade to a more current postgresql release. Being able to replay a days workload in a somewhat realistic manner would be a great help. -dg -- David Gould daveg@sonic.net 510 536 1443 510 282 0869 If simplicity worked, the world would be overrun with insects.
Greg Smith wrote: > > The Linux kernel developers are very clear that they don't care one > bit about testing for stability or lack of data loss in any > system-oriented way. That's somebody else's job now, typically the > Linux distributor who decides which of the kernels floating around are > the most stable, hopefully running more and larger tests than the > kernel developers do. > > Right. And we are not in any position to do that, even if we wanted to. The Linux kernel is a great example of how we don't want to do it, IMNSHO. We need to enhance our testing to preserve our reputation for stability and reliability, not throw the responsibility over the fence. cheers andrew
On Thu, Aug 27, 2009 at 08:02:03PM -0700, Ron Mayer wrote: > Andrew Dunstan wrote: > > I don't know of anyone who is likely to want to try out alphas in their > > normal development environments. The client I approached was > > specifically prepared to test beta releases that way. > > Perhaps end-users won't, but I think companies who develop software that > works on top of postgres will. Perhaps to make sure their existing software > continues to work; or perhaps to get a head start working with new features. > I test against CVS-head occasionally. I've been trying to help a client take up new versions of postgresql more quickly as the performance or feature content is often very valuable to them. Accordingly, I have encouraged them to run periodic samples of the nightly snapshots on at least one development instance, and to run the betas in the test environment. The goal is to be confident on the day of the postgresql release that we have tested enough and fixed any incompatibilities so that, if they so choose, they could migrate production to the new release immediately. This way they get several months extra benefit from improvements to postgresql. -dg -- David Gould daveg@sonic.net 510 536 1443 510 282 0869 If simplicity worked, the world would be overrun with insects.
On Fri, Aug 28, 2009 at 7:35 AM, Greg Smith<gsmith@gregsmith.com> wrote: > It's really amazing that a useful result ever comes out of this model at > all, and I know I'm not alone that I presume all Linux kernel releases are > too full of bugs to be useful until I've proven otherwise with my own QA. > > If the core PostgreSQL development worked like the Linux kernel, at the end > of each CommitFest whatever was done at that point would get published as > the new release. Instead of pausing to focus on a stable release everyone > would just speed ahead, backporting any major issues not discovered until > after the official release. The lesson that they took from earlier releases was that "pausing" development just led to heartache and delays and didn't actually help the release at all. Keep in mind that the Linux kernel itself is just an integration effort now anyways. All the actual development happens in other trees earlier anyways. Patches are only sent up to Linux when they've been fully developed (and hopefully somewhat tested) elsewhere. The arguments against the Linux approach are that a) it involves a lot of backpatching which is a pain. This is less convincing than it appears because the more often you fork branches the less different they all are. b) Our developers are also our testers and we don't have independent distribution vendors available to do testing. Actually we do, aside from people like EDB who have large test suites we're discussing how to get more testing from users precisely because our developers aren't really our testers. The big difference between Linux and ourselves is that it's a lot more work to migrate a database. So nobody would be particularly helped by having frequent releases. It would make a lot of sense even for us when the day comes that most people just run whatever Redhat or Debian ship. In which case we could come out with releases but users would happily ignore those releases until Redhat or Debian picked out, tested it, and released it in their distributions. -- greg http://mit.edu/~gsstark/resume.pdf
Greg Stark wrote: > They basically don't do any integration testing and leave that up to > the distributions now. Instead they have an "rc" release *every week* > like clockwork and every 2-3 months the last one becomes a new version > regardless of whether there's any notable new feature. They have a two week "merge period" during which all patches of any importance are merged. The RCs are released after that, and they are supposed to include only bugfixes to the merged patches. Some things are still merged after the merge window is closed, but they are very few. They release as many RCs as are necessary to close the majority of bugs/regressions. So yeah, they have some of the "time-based" nature, but they also use the "when it's ready" philosophy a lot. -- Alvaro Herrera http://www.CommandPrompt.com/ PostgreSQL Replication, Consulting, Custom Development, 24x7 support
Ron Mayer <rm_pg@cheapcomplexdevices.com> wrote: > Josh Berkus wrote: >> There's some very good reasons for the health of the project to >> have specific release dates and stick to them. > > Help me understand why? I don't know how many places are like this, but to get any significant staff or hardware resources officially allocated to anything here, you need a minimum of three months lead time. (Less, of course, if things are crashing and burning around our users' ears; more if the managers don't see an immediate and direct benefit to the users.) Any hope of organized participation by the Wisconsin Courts in a beta program would require a date they can put on their calendars and schedule around with confidence. As it is, what I do is based on having permission to run tests on my own time when there are hardware resources I can find to use which won't disrupt anything. From my perspective, a hard date for the beta release is more important than a hard date for the production release. Management here is very easy to sell on the concept that PostgreSQL stays in beta testing until there is confidence that the release is stable and trustworthy. -Kevin
All, >>> There's some very good reasons for the health of the project to have >>> specific release dates and stick to them. >> Help me understand why? We've cited this before, but here's the definitive paper on the subject: http://www.cyrius.com/publications/michlmayr-phd.pdf summary here: http://robertogaloppini.net/2007/04/02/open-source-production-time-based-release-management/ Further evidence since then with Debian, Parrot, and several other projects which have moved from feature-based releases to time-based releases is that each release had just as many features despite the more rigid time, and that contributors and users were far better able to organize their lives around a defined schedule. Thus, each person was able to contribute more because they knew exactly when they had to do it. I don't know about you, but I personally work better when I have an actual deadline. I'd think the advantages for our commercial adopters (who pay the salaries for many of the people on this list) would be obvious; if they know with a small margin of error when the next version of PostgreSQL is coming out, they can plan testing and deployment of their new products.See Kevin's post; many companies need to scheduleserious testing hardware months in advance, and every ISV needs to plan new product deployments up to a year in advance. We bitch a lot in the community about the super-old versions of PG which commercial software is using, but our variable release cycle is partly to blame. For our contributors, it's a lot harder to get funding to do paid contract work on features if you can't tell your sponsor when the next release is coming out. The alternative? 100% of the evidence I've seen is that feature-based release cycles get progressively longer as the project gets bigger. Eventually you find yourself only doing a release every 3.5 years, like MySQL or Debian used to. And your developers start leaving because they never see the fruits of their contributions, and your users start leaving because there's never anything new. Certainly our project experiences with "waiting for feature X" have all been negative. The windows port never got into 7.4 despite holding it up 4 months. HOT held up 8.3 for three to five months, depending on how you count it, in what I think everyone feels was our most painful beta period ever. Most recently, we let HS/SR hold up 8.4 for 2 months ... and they still weren't ready. I would like to see us go to an annual release timeline in which we release in the same month every year. Any time we say "variable release date" what it really means is "later release date". We've never yet released something *early*. -- Josh Berkus PostgreSQL Experts Inc. www.pgexperts.com
Folks, Here is my proposal for CFs for this year: We do four CFs, July 15, September 15, November 15, and January 15. However, we rigidly apply the 30-day deadline for the January 15 CF. That is, anything which is not completely ready for commit on February 14 gets punted to the next version. None of the "oh it's almost ready", no "just 2 more weeks of review". Make sure everything gets reviewed promptly (we *can* do it) and commit or punt. == Speeding things up == As far as our annual cycle is concerned, I don't think that the development/CF period is the problem; it's as efficient as we really want it to be. It's what comes after: post-CF cleanup, integration, beta. This period is especially a problem because it is one of little visible activity, no development, and generally waiting-room mentality for most of our contributors. Here are my proposals for making all of that go faster, with the caveat that it probably won't get better until the 2nd time we do it: Post-CF:Make a list (now, not in January) of the tasks which need to be done between CFend and Beta. We'll find that some of them could be done by someone other than Tom and Bruce, and that others could be done before CFend. Beta:Create a mailing list (why don't we have a list for testers? is testing less important than the JDBC driver?) and a simple web app or templated wiki page for testers. Allow people to check in with which version they tested (Alpha1,2,3, Beta 1,2,3) and what tests they ran, and issues encountered (if any). We should do this now so we can get started with Alpha testing.When this testing gets into swing, we can also look at recruiting volunteers to run various popular OSS apps' test suites against the developing version of PostgreSQL.Once beta starts, have a list of pending issues in some editable/commentable location (wiki is ok for this, or we can pervert the commitfest app) *as soon as those issues arise* so that as many hackers as possible can work on those issues. We did do a "pending issues" list for 8.4, but not until we were already over a month into beta, and then the list wasn't very accurate. -- Josh Berkus PostgreSQL Experts Inc. www.pgexperts.com
On Fri, 2009-08-28 at 10:55 -0700, Josh Berkus wrote: > Here is my proposal for CFs for this year: > > We do four CFs, July 15, September 15, November 15, and January 15. > > However, we rigidly apply the 30-day deadline for the January 15 CF. > That is, anything which is not completely ready for commit on February > 14 gets punted to the next version. None of the "oh it's almost ready", > no "just 2 more weeks of review". Make sure everything gets reviewed > promptly (we *can* do it) and commit or punt. Fine for me. -- Simon Riggs www.2ndQuadrant.com
Robert Haas wrote: > Both committers and non-committers are currently suffering from the > fact that there is not a lot of time in the year which is set aside > for development, i.e. neither CommitFest-time nor beta-time. To fix > this problem, we can: > > 1. Make CommitFests shorter. > 2. Make CommitFests less frequent. > 3. Continue developing during CommitFests. > 4. Make beta cycles shorter. > 5. Make beta cycles less frequent (i.e. lengthen the release cycle). > 6. Continue developing during beta. > > I believe (1) to be completely impractical and (3) to be > self-defeating. I suspect (2) will backfire badly. That doesn't > leave us with a lot of options. We can certainly do (5), but the > downside is that features that get committed won't hit release for a > very long time. I and others have suggested a couple of possible > approaches toward (4) or (6), such as changing the way we do release > notes, adding more regression tests to give us more (not perfect) > confidence that the release is solid, and/or branching the tree during > beta. None of those ideas have gotten a single vote of confidence > from you or Bruce. What's your suggestion? Another solution would be to make major releases less frequent. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. +
Tom Lane wrote: > "Kevin Grittner" <Kevin.Grittner@wicourts.gov> writes: > > Robert Haas <robertmhaas@gmail.com> wrote: > >> The final CommitFest began November 11, 2008. It closed March 25, > >> 2009 (+ 144 days). Beta1 was released April 15, 2009 (+ 21 days). > > > I'm not entirely clear on what was happening during the 21 days > > between the end of the CommitFest and and the release of beta1. > > Release note drafting is one part of it, but mostly it's "loose end > cleanup". Historically there have always been a pile of loose ends > to be dealt with, and the CommitFest structure doesn't really do > anything to avoid that. If you're interested, attached are all the > commits between commitfest closure (which I announced immediately > after committing the addition of contrib/btree_gin) and stamping > beta1 (which was actually several days before the date Robert gives, > because of the need for the packagers to do their thing). > > It appears to me that release notes weren't actually the bottleneck this > time, though they have been in some prior cycles. Bruce committed a > first draft immediately after the commitfest closed. Rather, it was > arguing about things like \df behavior and cardinality() that took up > the time. Yep, the bottom line here is that patches get into CVS, but issues come up related to the patch, and we keep looking for good fixes, but once the final commit-fest is over, we _have_ to fix these issues. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. +
Bruce, >> None of those ideas have gotten a single vote of confidence >> from you or Bruce. What's your suggestion? > > Another solution would be to make major releases less frequent. That's not a solution and you know it. Our development cycle has to change with the growth of the project. I know you don't like change and are comfortable with how we used to do things in 2001. But at this point the old practices are holding us back and we need to continue growing, or die. Our old development cycle was, effectively, single-process just like the old database engine was once. Making development more efficient and better for all contributors is largely a process of making it parallel by the incorporation of more people on every step, which also requires increased transparency, openness and tracking. Otherwise, like an overloaded database application in serializable mode, our development will just get slower and slower until it stops completely. -- Josh Berkus PostgreSQL Experts Inc. www.pgexperts.com
Josh Berkus wrote: > Bruce, > > >> None of those ideas have gotten a single vote of confidence > >> from you or Bruce. What's your suggestion? > > > > Another solution would be to make major releases less frequent. > > That's not a solution and you know it. I do? > Our development cycle has to change with the growth of the project. I > know you don't like change and are comfortable with how we used to do I don't? Wow, you are helping me see the light? > things in 2001. But at this point the old practices are holding us back > and we need to continue growing, or die. > > Our old development cycle was, effectively, single-process just like the > old database engine was once. Making development more efficient and > better for all contributors is largely a process of making it parallel > by the incorporation of more people on every step, which also requires > increased transparency, openness and tracking. > > Otherwise, like an overloaded database application in serializable mode, > our development will just get slower and slower until it stops completely. I have no idea how you know so much about me, but don't realize I was saying that we should extend the release cycle so we don't release as often, "make major releases less frequent" (every 12-14 months). This has nothing to do with how we process the releases, parallel or not. As I have said in the past, we are nearing feature-completeness (in a way), so having perhaps an 18-month release cycle is an idea. That would give more time for development compared to beta, etc. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. +
On Sat, Aug 29, 2009 at 1:05 PM, Bruce Momjian<bruce@momjian.us> wrote: > Robert Haas wrote: >> Both committers and non-committers are currently suffering from the >> fact that there is not a lot of time in the year which is set aside >> for development, i.e. neither CommitFest-time nor beta-time. To fix >> this problem, we can: >> >> 1. Make CommitFests shorter. >> 2. Make CommitFests less frequent. >> 3. Continue developing during CommitFests. >> 4. Make beta cycles shorter. >> 5. Make beta cycles less frequent (i.e. lengthen the release cycle). >> 6. Continue developing during beta. >> >> I believe (1) to be completely impractical and (3) to be >> self-defeating. I suspect (2) will backfire badly. That doesn't >> leave us with a lot of options. We can certainly do (5), but the >> downside is that features that get committed won't hit release for a >> very long time. I and others have suggested a couple of possible >> approaches toward (4) or (6), such as changing the way we do release >> notes, adding more regression tests to give us more (not perfect) >> confidence that the release is solid, and/or branching the tree during >> beta. None of those ideas have gotten a single vote of confidence >> from you or Bruce. What's your suggestion? > > Another solution would be to make major releases less frequent. I mentioned that one. It's #5. I also mentioned the downside. Downthread you make reference to PostgreSQL being feature-complete, but I don't personally think that's the right way to think about it. The people who are submitting patches are doing so because they feel that the product is missing the features implemented by those patches.Sure, some patches are pure performance plays, or bugfixes, or documentation improvements, but many of them are new features, plain and simple. And the people who submit those patches want to see them in a released version on a reasonable time scale. I do, anyway. I really can't understand why it isn't possible for us to find a way to make an annual release happen, and with more than 8-12 weeks of development time (ie non-CommitFest non-beta) available during that year. I understand that there is a need for some time to be set aside for reviewing, testing, debugging, packaging, and so forth, but the current schedule contemplates that this time is upwards of 75%, and I think that's excessive. ...Robert
>>> Another solution would be to make major releases less frequent. >> That's not a solution and you know it. > > I do? Ok, here's the reasons it's not a solution: 1) having a longer development cycle would frustrate many users who want new features sooner, not later. The current 1 year is a good compromise between reliability and "release often". A longer period would not be. 2) Lengthening the development period would make things less efficient.The amount of effort we need to test, document, integrate,package, etc., gets *greater* per patch when we have hundreds of patches. So if we *planned* an 18-month release, I expect that it would end up being a 24-month release. 3) If we deliberately lengthen the release cycle without doing anything about why the post-CF portion takes so long, it will continue to get longer, indefinitely. Eventually, we're at 3.5 year releases and our users abandoning Postgres for another database who can actually get a release out. 4) It does nothing to address the *contributor* complaint that the non-development part of our dev cycle is too long and keeps getting longer. A longer release cycle would make that worse. If we could concievably do a release every 4 months, I believe that it would be easy to keep the non-development portion of our cycle down to 30% or less. We can't, so we need to look at ways to speed up the work we're already doing. > I have no idea how you know so much about me, but don't realize I was > saying that we should extend the release cycle so we don't release as > often, "make major releases less frequent" (every 12-14 months). This > has nothing to do with how we process the releases, parallel or not. OK, to restate: making the cycle longer will not help the development-to-integration&testing ratio. It will make it worse. > As I have said in the past, we are nearing feature-completeness (in a > way), so having perhaps an 18-month release cycle is an idea. That > would give more time for development compared to beta, etc. Per the above, it would not. It would make things worse. This has been true at every other OSS project I've seen documented (disastrously so with MySQL); there is no reason to believe that Postgres would be any different. I also do not see why you are so resistant to the idea of documenting a tracking the post-CF steps so that we can get more people on them. -- Josh Berkus PostgreSQL Experts Inc. www.pgexperts.com
On Mon, 2009-08-31 at 10:30 -0700, Josh Berkus wrote: > >>> Another solution would be to make major releases less frequent. > >> That's not a solution and you know it. > > > > I do? > > Ok, here's the reasons it's not a solution: > Per the above, it would not. It would make things worse. This has been > true at every other OSS project I've seen documented (disastrously so > with MySQL); there is no reason to believe that Postgres would be any > different. > > I also do not see why you are so resistant to the idea of documenting a > tracking the post-CF steps so that we can get more people on them. > I love how we all have the same arguments, every year, year after year. So let me just throw in my proverbial two cents. As I see it we can *NOT* increase our development time line. Open Source just doesn't work that way. People want it, and want it now. Period. It will alienate feature contributors and make us fodder for bad press (blogs whatever) when we are lacking in some area where another isn't. We can decrease our development cycle. We could do an Ubuntu (and similarly Fedora) style cycle where people that want the hot new features now, can. They would do this by using our 6 month releases, while stable enterprises would use our LTS release. This is "kind of" happening now with our new Alpha release status. We can release annually and go all balls toward each release. The second option seems to be the middle ground that we will settle on regardless of what arguments are presented. The third option is what I would like to see happen. Which means we would actually have a 9 month development cycle/cutoff and a three month alpha/beta/release. Joshua D. Drake -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564 Consulting, Training, Support, Custom Development, Engineering
Bruce Momjian <bruce@momjian.us> wrote: > Yep, the bottom line here is that patches get into CVS, but issues > come up related to the patch, and we keep looking for good fixes, > but once the final commit-fest is over, we _have_ to fix these > issues. If, hypothetically, it might hold up the release for two weeks while such issues are sorted out, might it be better to revert and say the patch missed the release because it wasn't workable enough at the end of the last CF to allow a beta release to be generated? If the net result was that a feature or two were delayed until the next release, but all developers had two more weeks of development time in the next release cycle, it seems like reverting would be a net gain. -Kevin
Josh Berkus wrote: > Per the above, it would not. It would make things worse. This has been > true at every other OSS project I've seen documented (disastrously so > with MySQL); there is no reason to believe that Postgres would be any > different. > > I also do not see why you are so resistant to the idea of documenting a > tracking the post-CF steps so that we can get more people on them. Huh, who has asked for a list from me? This entire post is mostly over-the-top and not worth responding to. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. +
On Mon, Aug 31, 2009 at 1:57 PM, Bruce Momjian<bruce@momjian.us> wrote: > Josh Berkus wrote: >> Per the above, it would not. It would make things worse. This has been >> true at every other OSS project I've seen documented (disastrously so >> with MySQL); there is no reason to believe that Postgres would be any >> different. >> >> I also do not see why you are so resistant to the idea of documenting a >> tracking the post-CF steps so that we can get more people on them. > > Huh, who has asked for a list from me? This entire post is mostly > over-the-top and not worth responding to. OK, I so request. :-) ...Robert
Kevin Grittner wrote: > Bruce Momjian <bruce@momjian.us> wrote: > > > Yep, the bottom line here is that patches get into CVS, but issues > > come up related to the patch, and we keep looking for good fixes, > > but once the final commit-fest is over, we _have_ to fix these > > issues. > > If, hypothetically, it might hold up the release for two weeks while > such issues are sorted out, might it be better to revert and say the > patch missed the release because it wasn't workable enough at the end > of the last CF to allow a beta release to be generated? If the net > result was that a feature or two were delayed until the next release, > but all developers had two more weeks of development time in the next > release cycle, it seems like reverting would be a net gain. The problem is that many of these decisions are complex so it gets no easier to make the decisions later rather than now. The delay forces us to make a final decision. We often had months to make the decision earlier, but didn't. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. +
Robert Haas wrote: > On Mon, Aug 31, 2009 at 1:57 PM, Bruce Momjian<bruce@momjian.us> wrote: > > Josh Berkus wrote: > >> Per the above, it would not. ?It would make things worse. ?This has been > >> true at every other OSS project I've seen documented (disastrously so > >> with MySQL); there is no reason to believe that Postgres would be any > >> different. > >> > >> I also do not see why you are so resistant to the idea of documenting a > >> tracking the post-CF steps so that we can get more people on them. > > > > Huh, who has asked for a list from me? ?This entire post is mostly > > over-the-top and not worth responding to. > > OK, I so request. :-) What do you want to know? Would someone post exactly what question I have not answered in the past? -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. +
On Mon, Aug 31, 2009 at 1:59 PM, Bruce Momjian<bruce@momjian.us> wrote: > Robert Haas wrote: >> On Mon, Aug 31, 2009 at 1:57 PM, Bruce Momjian<bruce@momjian.us> wrote: >> > Josh Berkus wrote: >> >> Per the above, it would not. ?It would make things worse. ?This has been >> >> true at every other OSS project I've seen documented (disastrously so >> >> with MySQL); there is no reason to believe that Postgres would be any >> >> different. >> >> >> >> I also do not see why you are so resistant to the idea of documenting a >> >> tracking the post-CF steps so that we can get more people on them. >> > >> > Huh, who has asked for a list from me? ?This entire post is mostly >> > over-the-top and not worth responding to. >> >> OK, I so request. :-) > > What do you want to know? Would someone post exactly what question I > have not answered in the past? I don't know whether there is a specific question that you have refused to answer in the past, or not. My suspicion is that there isn't, but perhaps someone else is aware of something I'm not. That having been said, I think there is a legitimate concern about organizing and documenting the steps that are required to get a release out the door. A number of people have said (on this thread and previous ones) that we didn't know what we were supposed to be doing during the period after the end of the last CommitFest and prior to release. It appeared that all of the activity (to the extent that there was any activity) was by committers, particularly you and Tom. Well, OK. We want the release to happen faster next time. We're willing to help. In order to help, we first need a list of the tasks that need to be completed after the last CommitFest and before release. Then we can try to figure out whether any of those tasks can be done (or assisted with) by someone other than you or Tom. Can you provide one? ...Robert
On Mon, 2009-08-31 at 13:59 -0400, Bruce Momjian wrote: > Robert Haas wrote: > > On Mon, Aug 31, 2009 at 1:57 PM, Bruce Momjian<bruce@momjian.us> wrote: > > > Josh Berkus wrote: > > >> Per the above, it would not. ?It would make things worse. ?This has been > > >> true at every other OSS project I've seen documented (disastrously so > > >> with MySQL); there is no reason to believe that Postgres would be any > > >> different. > > >> > > >> I also do not see why you are so resistant to the idea of documenting a > > >> tracking the post-CF steps so that we can get more people on them. > > > > > > Huh, who has asked for a list from me? ?This entire post is mostly > > > over-the-top and not worth responding to. > > > > OK, I so request. :-) > > What do you want to know? Would someone post exactly what question I > have not answered in the past? This is a fair point. I bet 10 bucks that a lot of the questions that would be asked would be answered with, "check the archives". Didn't we do a release wiki page at one point? Joshua D. Drake -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564 Consulting, Training, Support, Custom Development, Engineering
Bruce, > Huh, who has asked for a list from me? This entire post is mostly > over-the-top and not worth responding to. To quote myself: > Post-CF: > Make a list (now, not in January) of the tasks which need to be done > between CFend and Beta. We'll find that some of them could be done by > someone other than Tom and Bruce, and that others could be done before > CFend. > > Beta: > Create a mailing list (why don't we have a list for testers? is > testing less important than the JDBC driver?) and a simple web app or > templated wiki page for testers. Allow people to check in with which > version they tested (Alpha1,2,3, Beta 1,2,3) and what tests they ran, > and issues encountered (if any). We should do this now so we can get > started with Alpha testing. > When this testing gets into swing, we can also look at recruiting > volunteers to run various popular OSS apps' test suites against the > developing version of PostgreSQL. > Once beta starts, have a list of pending issues in some > editable/commentable location (wiki is ok for this, or we can pervert > the commitfest app) *as soon as those issues arise* so that as many > hackers as possible can work on those issues. We did do a "pending > issues" list for 8.4, but not until we were already over a month into > beta, and then the list wasn't very accurate. Therefore: I will create a "cleanup issues" wikipage. Please contribute to it by listing the *general* kinds of things you need to do between CF and beta. Then we can look at which things don't need your special experience and could be done by other contributors. -- Josh Berkus PostgreSQL Experts Inc. www.pgexperts.com
Robert Haas wrote: > That having been said, I think there is a legitimate concern about > organizing and documenting the steps that are required to get a > release out the door. A number of people have said (on this thread > and previous ones) that we didn't know what we were supposed to be > doing during the period after the end of the last CommitFest and prior > to release. It appeared that all of the activity (to the extent that > there was any activity) was by committers, particularly you and Tom. > > Well, OK. We want the release to happen faster next time. We're > willing to help. In order to help, we first need a list of the tasks > that need to be completed after the last CommitFest and before > release. Then we can try to figure out whether any of those tasks can > be done (or assisted with) by someone other than you or Tom. > > Can you provide one? Well, at the end of the release I have a mailbox full of open items, that I think need to be addressed before we go into beta. Tom has a similar list. I usually put my mbox file up on a web site, and sometimes it is transfered to a wiki by others. Knowing about the problem usually isn't hard , e.g. \df, but getting agreement on them is. One nifty idea would be to do a commit-fest for open items so we can get to beta. The last commit-fest usually is long because we can't postpone patches easily and often we are not 100% sure how to apply them either, so that make it extra-long. I am not sure what other checklist items there would be (or I am refusing to divulge). -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. +
Joshua D. Drake wrote: > On Mon, 2009-08-31 at 13:59 -0400, Bruce Momjian wrote: > > Robert Haas wrote: > > > On Mon, Aug 31, 2009 at 1:57 PM, Bruce Momjian<bruce@momjian.us> wrote: > > > > Josh Berkus wrote: > > > >> Per the above, it would not. ?It would make things worse. ?This has been > > > >> true at every other OSS project I've seen documented (disastrously so > > > >> with MySQL); there is no reason to believe that Postgres would be any > > > >> different. > > > >> > > > >> I also do not see why you are so resistant to the idea of documenting a > > > >> tracking the post-CF steps so that we can get more people on them. > > > > > > > > Huh, who has asked for a list from me? ?This entire post is mostly > > > > over-the-top and not worth responding to. > > > > > > OK, I so request. :-) > > > > What do you want to know? Would someone post exactly what question I > > have not answered in the past? > > This is a fair point. I bet 10 bucks that a lot of the questions that > would be asked would be answered with, "check the archives". > > Didn't we do a release wiki page at one point? Yes, we have a wiki for open items for the current major release. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. +
Josh Berkus wrote: > Bruce, > > > Huh, who has asked for a list from me? This entire post is mostly > > over-the-top and not worth responding to. > > To quote myself: > > > Post-CF: > > Make a list (now, not in January) of the tasks which need to be done > > between CFend and Beta. We'll find that some of them could be done by > > someone other than Tom and Bruce, and that others could be done before > > CFend. > > > > Beta: > > Create a mailing list (why don't we have a list for testers? is > > testing less important than the JDBC driver?) and a simple web app or > > templated wiki page for testers. Allow people to check in with which > > version they tested (Alpha1,2,3, Beta 1,2,3) and what tests they ran, > > and issues encountered (if any). We should do this now so we can get > > started with Alpha testing. > > When this testing gets into swing, we can also look at recruiting > > volunteers to run various popular OSS apps' test suites against the > > developing version of PostgreSQL. > > Once beta starts, have a list of pending issues in some > > editable/commentable location (wiki is ok for this, or we can pervert > > the commitfest app) *as soon as those issues arise* so that as many > > hackers as possible can work on those issues. We did do a "pending > > issues" list for 8.4, but not until we were already over a month into > > beta, and then the list wasn't very accurate. > > Therefore: > > I will create a "cleanup issues" wikipage. Please contribute to it by > listing the *general* kinds of things you need to do between CF and > beta. Then we can look at which things don't need your special > experience and could be done by other contributors. That was a request for me to answer? I had no idea. The issues are different for every commitfest-beta period, so I have no idea what to list there, but we do alway have an open issues wiki that is maintained, at least for the most recent releases. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. +
On Mon, Aug 31, 2009 at 2:20 PM, Bruce Momjian<bruce@momjian.us> wrote: > Knowing about the problem usually isn't hard , e.g. \df, but getting > agreement on them is. One nifty idea would be to do a commit-fest for > open items so we can get to beta. I like that idea very much. > The last commit-fest usually is long > because we can't postpone patches easily and often we are not 100% sure > how to apply them either, so that make it extra-long. > > I am not sure what other checklist items there would be (or I am > refusing to divulge). LOL. Well, there are things like release notes... and maybe others? ...Robert
Bruce Momjian <bruce@momjian.us> wrote: > The issues are different for every commitfest-beta period, so I have > no idea what to list there, but we do alway have an open issues wiki > that is maintained, at least for the most recent releases. After a quick search of the wiki, it appears that the list for 8.4 was: http://wiki.postgresql.org/wiki/PostgreSQL_8.4_Open_Items and that there is not yet a list for 8.5. Is that correct? If I understand what you're saying, this list would contain issues where a patch was committed and later found to have problems which need to be fixed. Did I understand that correctly? Anything else go on there, or possibly belong on there? Can we take the absence of a list for 8.5 to indicate that no such problems have been found with any patches committed since 8.4 was tagged? -Kevin
Bruce Momjian <bruce@momjian.us> wrote: > it gets no easier to make the decisions later rather than now. The > delay forces us to make a final decision. We often had months to > make the decision earlier, but didn't. So you're advocating that we find a way to force more timely decisions? -Kevin
On sön, 2009-08-30 at 21:09 -0400, Robert Haas wrote: > I really can't understand why it isn't possible for us to find a way > to make an annual release happen, and with more than 8-12 weeks of > development time (ie non-CommitFest non-beta) available during that > year. I understand that there is a need for some time to be set aside > for reviewing, testing, debugging, packaging, and so forth, but the > current schedule contemplates that this time is upwards of 75%, and I > think that's excessive. Well, the best way to improve that is to organize people and push things forward when the time comes.
Kevin Grittner wrote: > Bruce Momjian <bruce@momjian.us> wrote: > > > it gets no easier to make the decisions later rather than now. The > > delay forces us to make a final decision. We often had months to > > make the decision earlier, but didn't. > > So you're advocating that we find a way to force more timely > decisions? That would be good. :-) -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. +
Kevin Grittner wrote: > Bruce Momjian <bruce@momjian.us> wrote: > > > The issues are different for every commitfest-beta period, so I have > > no idea what to list there, but we do alway have an open issues wiki > > that is maintained, at least for the most recent releases. > > After a quick search of the wiki, it appears that the list for 8.4 > was: > > http://wiki.postgresql.org/wiki/PostgreSQL_8.4_Open_Items > > and that there is not yet a list for 8.5. Is that correct? > > If I understand what you're saying, this list would contain issues > where a patch was committed and later found to have problems which > need to be fixed. Did I understand that correctly? Anything else go > on there, or possibly belong on there? Can we take the absence of a > list for 8.5 to indicate that no such problems have been found with > any patches committed since 8.4 was tagged? Yes, though I have a few items that I should transfer from my mailbox to that list. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. +
Bruce, > I am not sure what other checklist items there would be (or I am > refusing to divulge). Hopefully the last is a joke. ;-) So, the only post-CF tasks are issues with specific patches? This seems resolvable, especially if we take a hard line with patch readiness. There isn't anything else in that period? This doesn't sound that difficult then. -- Josh Berkus PostgreSQL Experts Inc. www.pgexperts.com
Josh Berkus wrote: > Bruce, > > > I am not sure what other checklist items there would be (or I am > > refusing to divulge). > > Hopefully the last is a joke. ;-) Yes. > So, the only post-CF tasks are issues with specific patches? This seems > resolvable, especially if we take a hard line with patch readiness. > There isn't anything else in that period? Nope. Release notes and open items is all I remember, and the release notes were done at the end of the commit-fest, so that isn't really even an issue. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. +
On Mon, 2009-08-31 at 10:30 -0700, Josh Berkus wrote: > >>> Another solution would be to make major releases less frequent. > >> That's not a solution and you know it. > > > > I do? > > Ok, here's the reasons it's not a solution: > Per the above, it would not. It would make things worse. This has been > true at every other OSS project I've seen documented (disastrously so > with MySQL); there is no reason to believe that Postgres would be any > different. > > I also do not see why you are so resistant to the idea of documenting a > tracking the post-CF steps so that we can get more people on them. > I love how we all have the same arguments, every year, year after year. So let me just throw in my proverbial two cents. As I see it we can *NOT* increase our development time line. Open Source just doesn't work that way. People want it, and want it now. Period. It will alienate feature contributors and make us fodder for bad press (blogs whatever) when we are lacking in some area where another isn't. We can decrease our development cycle. We could do an Ubuntu (and similarly Fedora) style cycle where people that want the hot new features now, can. They would do this by using our 6 month releases, while stable enterprises would use our LTS release. This is "kind of" happening now with our new Alpha release status. We can release annually and go all balls toward each release. The second option seems to be the middle ground that we will settle on regardless of what arguments are presented. The third option is what I would like to see happen. Which means we would actually have a 9 month development cycle/cutoff and a three month alpha/beta/release. Joshua D. Drake -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564 Consulting, Training, Support, Custom Development, Engineering
On Mon, 2009-08-31 at 13:59 -0400, Bruce Momjian wrote: > Robert Haas wrote: > > On Mon, Aug 31, 2009 at 1:57 PM, Bruce Momjian<bruce@momjian.us> wrote: > > > Josh Berkus wrote: > > >> Per the above, it would not. ?It would make things worse. ?This has been > > >> true at every other OSS project I've seen documented (disastrously so > > >> with MySQL); there is no reason to believe that Postgres would be any > > >> different. > > >> > > >> I also do not see why you are so resistant to the idea of documenting a > > >> tracking the post-CF steps so that we can get more people on them. > > > > > > Huh, who has asked for a list from me? ?This entire post is mostly > > > over-the-top and not worth responding to. > > > > OK, I so request. :-) > > What do you want to know? Would someone post exactly what question I > have not answered in the past? This is a fair point. I bet 10 bucks that a lot of the questions that would be asked would be answered with, "check the archives". Didn't we do a release wiki page at one point? Joshua D. Drake -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564 Consulting, Training, Support, Custom Development, Engineering
On Thu, Aug 27, 2009 at 12:03:05AM +0200, Dimitri Fontaine wrote: > Hi, > > Peter Eisentraut <peter_e@gmx.net> writes: > > On ons, 2009-08-26 at 14:26 -0400, Robert Haas wrote: > >> Sure, but an aimless mandate to do testing for 4 (or 8, or 12) months > >> doesn't necessarily buy you much, either. I'm good at focused > >> activity - but there was nothing focused about 8.4 beta that I could > >> see. Maybe we need some kind of TestFest process. > > > > Yeah, exactly. I can't imagine end users would know what to do during > > beta. Even assuming that you have release notes at the beginning of > > beta, you can't expect people to go through every item and do a formal > > test for it. Surely it's been tested before, else it would not be in > > the release, right? > > Well we all know that developpers are really bad at testing, because > they tend to test for cases they though about while developping the code > rather than being creative and running a full application against the > overall new product. > > Now, it could be that what we miss is some tool suite to enable > application people to test their full code against our new releases and > check results, performance, plans, etc. > > I know about a couple of tools to get there, Tsung and Playr. And the > focus is performance testing and scalability, not that it still works. > > Is the offering good enough? We might need to run some kind of tutorials > for users to be able to run large tests easily, and maybe think about > some newer tools allowing to compare logs of two application runs in two > database versions (capture all queries and result in a database, then > have a way to diff). Then beta testing would mean having a spare machine > where to run the magic regression test suite against some internal > application. Someone else upthread mentioned it as well; having an easy way to capture queries for a day and replaying it towards a server with the beta software would be a totally awesome way to beta test. Query results would be compared and any anomalies reported. Doing this in realtime over both servers would be an even more appealing option (at least to me and my environment). I could easily capture queries from a dev machine (where development is still towards a stable release of postgres) and test them on the new Beta release. I haven't looked at either Tsung or Playr (but I will), so if anyone knows if this is possible with these tools or perhaps is already doing it, it would be nice with a short "tutorial" (documented on the wiki?) on steps on how to setup such an environment. Kind regards, Kristian. -- Kristian Larsson KLL-RIPE +46 704 264511 kll@spritelink.net
On Sat, Aug 29, 2009 at 12:19 AM, Josh Berkus<josh@agliodbs.com> wrote: > I'd think the advantages for our commercial adopters (who pay the > salaries for many of the people on this list) would be obvious; if they > know with a small margin of error when the next version of PostgreSQL is > coming out, they can plan testing and deployment of their new products. > See Kevin's post; many companies need to schedule serious testing > hardware months in advance, and every ISV needs to plan new product > deployments up to a year in advance. We bitch a lot in the community > about the super-old versions of PG which commercial software is using, > but our variable release cycle is partly to blame. It also works on the other end - with time based releases you can also schedule obsolescence. It is just as critical knowing when the community will stop bug fixes and security fixes when you are trying to schedule major rollouts and planning product development. Canonical (my employer) certainly believe in time based releases, and that is one of the major reasons for the growth of Ubuntu and the Ubuntu Community. We now use time based releases for almost all our sponsored projects (some 6 monthly, some monthly), and are lobbying various projects and other OS distributions to get into some sort of cadence with releases so everyone benefits. It makes us happier (especially when we are choosing what we can commit to providing security updates for the 5 year releases), and our users happier, and I think you happier with less support issues. (In fact the one project I'm personally aware of that doesn't have time based releases also has the worst reputation for bug fixes and updates and caused us trauma because of it, so I'll be pushing to get that fixed too :-P) > Certainly our project experiences with "waiting for feature X" have all > been negative. The windows port never got into 7.4 despite holding it > up 4 months. HOT held up 8.3 for three to five months, depending on how > you count it, in what I think everyone feels was our most painful beta > period ever. Most recently, we let HS/SR hold up 8.4 for 2 months ... > and they still weren't ready. > > I would like to see us go to an annual release timeline in which we > release in the same month every year. Any time we say "variable release > date" what it really means is "later release date". We've never yet > released something *early*. Yes please. You may even want to seriously consider shorter release cycles. Tighter cycles can actually reduce stress, as people are less concerned with slippage. With our projects on one month cycles, it doesn't matter that much if a feature isn't good enough for a release - it just goes out with the next months release or the one after if you really underestimated the work. With longer cycles, the penalties of missing deadlines is much greater which can lead to cutting corners if people are not disciplined. Of course, PG already has its own historical cadence to start from where as we had the luxury of adopting time based releases at the start or relatively early in development. For PostgreSQL, with the regular commit fests you might end up to a similar process to GNU Bazaar except with yearly major releases and 2 month development releases, documented at http://doc.bazaar-vcs.org/latest/developers/cycle.html. This is a smaller project, but had to address a number of similar concerns that PostgreSQL would have to so may be a good basis for discussion. -- Stuart Bishop <stuart@stuartbishop.net> http://www.stuartbishop.net/
Stuart Bishop wrote: > Canonical (my employer) certainly believe in time based releases, and > that is one of the major reasons for the growth of Ubuntu and the > Ubuntu Community. We now use time based releases for almost all our > sponsored projects (some 6 monthly, some monthly), and are lobbying > various projects and other OS distributions to get into some sort of > cadence with releases so everyone benefits. It makes us happier > (especially when we are choosing what we can commit to providing > security updates for the 5 year releases), and our users happier, and > I think you happier with less support issues. > > The release cycle is quite independent of the release lifetime. In any case, I don't accept this analogy. The mechanics of a Linux distribution are very different from the mechanics of a project like PostgreSQL. The prominent OSS project that seems to me most like ours is the Apache HTTP project. But they don't do timed releases AFAIK, and theirs is arguably the most successful OSS project ever. I'm especially resistant to suggestions that we should in some way coordinate our releases with other projects' timings. Getting our own developers organized is sufficiently like herding cats that I have no confidence that anyone will successfully organize those of a plethora of projects. I am not saying timed releases are necessarily bad. But many of the arguments that have been put forward to support them don't seem to me to withstand critical analysis. I would argue that it would be an major setback for us if we made another release without having Hot Standby or whatever we are calling it now. I would much rather slip one month or three than ship without it. cheers andrew
Andrew Dunstan <andrew@dunslane.net> writes:
> I would argue that it would be an major setback for us if we made
> another release without having Hot Standby or whatever we are calling it
> now. I would much rather slip one month or three than ship without it.
I would agree with you, except that every single time we have done that,
it's led to a scheduling disaster: the release was invariably far later
than anyone expected, and more often than not shipped without the
"critical" feature anyhow. Review the archives.
regards, tom lane
On Tue, Sep 8, 2009 at 7:54 PM, Andrew Dunstan<andrew@dunslane.net> wrote: > The release cycle is quite independent of the release lifetime. If you have dates on releases, it is easier to set dates on release lifetime. If you know the releases come out once a year at about the same time, and you want to have a set number of versions in play, you can state at release time when the community will stop support. This gives everyone a clear picture to people what versions they should be targeting and when upgrades will be required. > In any case, I don't accept this analogy. The mechanics of a Linux > distribution are very different from the mechanics of a project like > PostgreSQL. The prominent OSS project that seems to me most like ours is the > Apache HTTP project. But they don't do timed releases AFAIK, and theirs is > arguably the most successful OSS project ever. We find it works for stuff other than Ubuntu too. IIRC original concerns where you could do it for a small open source project, but it would be impossible to do when juggling as many moving parts as a Linux distribution. You might find the document I cited is for a project with similar issues to PostgreSQL and may address your concerns. It seems to work for other large projects too, such as Gnome, as well as smaller ones. People are discussing switching for reasons Joshua cited (maintaining momentum, planning, enterprise adoption etc.), because people find it a good idea on other projects they work with, or maybe because they read too many articles on agile and lean development practices. It seems to be working fine for me personally (I work on launchpad.net, which is an Open Source mostly-web application using generally Lean/Agile development methodologies, a one month release cycle and a team of about 30 spread over all timezones). > I'm especially resistant to suggestions that we should in some way > coordinate our releases with other projects' timings. Getting our own > developers organized is sufficiently like herding cats that I have no > confidence that anyone will successfully organize those of a plethora of > projects. I tend to think it will evolve naturally as more people switch to time based releases. Its natural to sync in with the OS releases your developers care about because it makes their lives easier, and its natural for the distributions to get in sync too because it makes their developer's lives easier. But only hindsight will tell of course :-) With a yearly schedule, it probably doesn't matter much except for distributions with a 2 or 3 year cycle - you would still end up with latest PostgreSQL a maximum of I think 8 months after the official release. > I am not saying timed releases are necessarily bad. But many of the > arguments that have been put forward to support them don't seem to me to > withstand critical analysis. > > I would argue that it would be an major setback for us if we made another > release without having Hot Standby or whatever we are calling it now. I > would much rather slip one month or three than ship without it. This is why you want your cycle as small as possible - if you have a 6 month cycle for instance, the feature would be available a maximum of 6 months after it is ready. With the feature based release cycle, what if it still isn't ready for prime time after three months of slippage? Having one feature slip hurts, but having all features slip hurts more. Josh cited several examples where he felt similar situations had hurt PostgreSQL development. Of course, if you think it is critical enough you can let it slip and if it is critical enough people will understand - we let one of the 10 Ubuntu releases slip once and people generally understood (you want to get a LTS release right since you have to live with your mistakes for 5 years). There was some flak but we are still here. I personally suspect PostgreSQL would want a 1 year cycle for major releases while a full dump/reload is required for upgrades. When this changes, 6 or even 4 months might actually be a good fit. -- Stuart Bishop <stuart@stuartbishop.net> http://www.stuartbishop.net/
> > I personally suspect PostgreSQL would want a 1 year cycle for major > releases while a full dump/reload is required for upgrades. When this > changes, 6 or even 4 months might actually be a good fit. > For some DBA specialist is 1 year cycle too much fast. I thing, so 1 year cycle is perfect for databases. Migration to new database releases is some different then migration to new program. You have to be more carefully, because wrong database could to destroy your data. Normal use case for using a new version contains one year waiting. I afraid so too much short cycle can break clean postgresql's release process. regards Pavel Stehule
It would be nice if we could enable and disable column and table constraints. I believe that you can do this in Oracle but this is very handy for testing stored procedures and other external processes. Best Regards -- Michael Gould, Managing Partner Intermodal Software Solutions, LLC 904.226.0978 904.592.5250 fax
> In any case, I don't accept this analogy. The mechanics of a Linux > distribution are very different from the mechanics of a project like > PostgreSQL. The prominent OSS project that seems to me most like ours is > the Apache HTTP project. But they don't do timed releases AFAIK, and > theirs is arguably the most successful OSS project ever. I can't find information about HTTPD release planning so I'll take your word for it. On the other hand, I have to point out that Apache is releasing HTTPD major versions an average of once every 3 years. I don't think we want to go to 3 years, do we? > I'm especially resistant to suggestions that we should in some way > coordinate our releases with other projects' timings. Agreed, that's impossible. I'd prefer just to time things so that we're not trying to do committer-critical tasks during high summer vacation season. > I am not saying timed releases are necessarily bad. But many of the > arguments that have been put forward to support them don't seem to me to > withstand critical analysis. There are several reasons to have a timed release schedule, which are detailed in the paper I linked. The benefits of having timed releases are for our contributors and for our users. > I would argue that it would be an major setback for us if we made > another release without having Hot Standby or whatever we are calling it > now. I would much rather slip one month or three than ship without it. Historically, any feature which hasn't been ready for the originally planned release date needed far more than an extra month to be ready -- up to an additional year. HOT is the only exception I can think of to this in our 13-year history. I agree that HS is critical to adoption of PostgreSQL. However, the way to make it succeed is to have deadlines and meet them. Not work on it "until it's ready". >> I personally suspect PostgreSQL would want a 1 year cycle for major >> releases while a full dump/reload is required for upgrades. When this >> changes, 6 or even 4 months might actually be a good fit. >> > > For some DBA specialist is 1 year cycle too much fast. I thing, so 1 > year cycle is perfect for databases. Migration to new database Pavel is right. Upgrade-in-place will become easier, but it will never become painless. So DBAs will always want to put off upgrades to once in 2-3 years. If we went to releases ever 6 months, that would mean users skipping 5 versions. Also, we backpatch fixes a lot more than Ubuntu does AFAIK. So we'd still have to patch back 5 years, which would be 10 versions ... I don't think so. Once/year is the right length for us. It's 2x to 3x faster than any other mission-critical DBMS. -- Josh Berkus PostgreSQL Experts Inc. www.pgexperts.com
Andrew Dunstan wrote: > In any case, I don't accept this analogy. The mechanics of a Linux > distribution are very different from the mechanics of a project like > PostgreSQL. The prominent OSS project that seems to me most like ours is > the Apache HTTP project. I'd think that File Systems might be more like postgres - with a shared obsession about data loss risks, and concerns about compatibility with any on-disk format changes. I wonder if the ext4 or btrfs guys use time-based release schedules, or if they'll release when it's ready. I see the ZFS guys have target dates for completing features that are still in beta, but also that they change as needed.[1] [1] http://opensolaris.org/os/project/zfs-crypto/ Anyone know how the F/OSS filesystem guys schedule their releases? I agree it's quite different than a distro - which, if I understand correctly, is mostly a matter of identifying completed and stable features rather than completing and stabilizing features. > I would argue that it would be an major setback for us if we made > another release without having Hot Standby or whatever we are calling it > now. I would much rather slip one month or three than ship without it. Perhaps if sufficiently interesting features get in outside of a time-based schedule, an extra release could be made after the commit fest it gets in? If hot-standby + streaming-replication + index_only_scans + magic-fairy-dust-powered-shared-nothing-clusters all happened to get in 3 months after a time-based release, it'd be nice to see it sooner rather than waiting 9 months for a time-based window.
Josh Berkus wrote: > I can't find information about HTTPD release planning so I'll take your > word for it. On the other hand, I have to point out that Apache is > releasing HTTPD major versions an average of once every 3 years. I > don't think we want to go to 3 years, do we? I'd say it depends on the flexibility of some hypothetical future module layer. If I understand right, much of the functionality in apache comes from modules - and those modules that are under heavy development may have different release cycles. I realize this doesn't work for many of the big features; but some of those seem to have over 1 year development cycles anyway.
On Tue, Sep 8, 2009 at 2:06 PM, Ron Mayer<rm_pg@cheapcomplexdevices.com> wrote: > Andrew Dunstan wrote: >> In any case, I don't accept this analogy. The mechanics of a Linux >> distribution are very different from the mechanics of a project like >> PostgreSQL. The prominent OSS project that seems to me most like ours is >> the Apache HTTP project. > > I'd think that File Systems might be more like postgres - with a shared > obsession about data loss risks, and concerns about compatibility > with any on-disk format changes. I wonder if the ext4 or btrfs > guys use time-based release schedules, or if they'll release when > it's ready. I see the ZFS guys have target dates for completing > features that are still in beta, but also that they change as needed.[1] > [1] http://opensolaris.org/os/project/zfs-crypto/ > > Anyone know how the F/OSS filesystem guys schedule their releases? > > > I agree it's quite different than a distro - which, if I understand > correctly, is mostly a matter of identifying completed and stable > features rather than completing and stabilizing features. > >> I would argue that it would be an major setback for us if we made >> another release without having Hot Standby or whatever we are calling it >> now. I would much rather slip one month or three than ship without it. > > Perhaps if sufficiently interesting features get in outside of > a time-based schedule, an extra release could be made after the > commit fest it gets in? > > If hot-standby + streaming-replication + index_only_scans + > magic-fairy-dust-powered-shared-nothing-clusters all happened > to get in 3 months after a time-based release, it'd be nice to > see it sooner rather than waiting 9 months for a time-based window. That's somewhat true, but major patches are also more likely to come with bugs. I think we ought to try to time major patches near the beginning of the release cycle, not the end. Indeed, I'd be much more inclined to support a proposal to branch the tree and do an out-of-sequence release just BEFORE committing a bunch of major features rather than just after. Otherwise, PostgreSQL's reputation for being a solid product with solid releases will suffer. ...Robert
Michael Gould <mgould@intermodalsoftwaresolutions.net> writes:
> It would be nice if we could enable and disable column and table
> constraints. I believe that you can do this in Oracle but this is very
> handy for testing stored procedures and other external processes.
Drop the constraint and re-add it later...
regards, tom lane
Tom Lane wrote: > Michael Gould <mgould@intermodalsoftwaresolutions.net> writes: > > It would be nice if we could enable and disable column and table > > constraints. I believe that you can do this in Oracle but this is very > > handy for testing stored procedures and other external processes. > > Drop the constraint and re-add it later... That's not very useful when adding it later means grabbing an exclusive lock on the table for the whole duration of the full table scan required to check the table. Actually something in this area is on my plate too -- a customer of ours wants to be able to define constraints but not have it checked immediately. I envision it similar to how concurrent index creation works: the constraint is created as "not checked" and the transaction is committed; new insertions are checked against the constraint. Then the table is scanned to verify that extant tuples pass the constraint, _without_ the exclusive lock on the table. Both DB2 and Oracle have an ENFORCE setting for constraints, and a MySQL blog hinted some time ago that it might be in SQL 201x. -- Alvaro Herrera http://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc.
On Tue, 2009-09-08 at 16:07 -0400, Alvaro Herrera wrote: > Tom Lane wrote: > > Michael Gould <mgould@intermodalsoftwaresolutions.net> writes: > > > It would be nice if we could enable and disable column and table > > > constraints. I believe that you can do this in Oracle but this is very > > > handy for testing stored procedures and other external processes. > > > > Drop the constraint and re-add it later... > > That's not very useful when adding it later means grabbing an exclusive > lock on the table for the whole duration of the full table scan required > to check the table. It's also useful to define foreign keys for documentation purposes but not enforce them for some reason.
On Tue, Sep 8, 2009 at 1:07 PM, Alvaro Herrera<alvherre@commandprompt.com> wrote: > Tom Lane wrote: >> Michael Gould <mgould@intermodalsoftwaresolutions.net> writes: >> > It would be nice if we could enable and disable column and table >> > constraints. I believe that you can do this in Oracle but this is very >> > handy for testing stored procedures and other external processes. >> >> Drop the constraint and re-add it later... > > That's not very useful when adding it later means grabbing an exclusive > lock on the table for the whole duration of the full table scan required > to check the table. > > Actually something in this area is on my plate too -- a customer of ours > wants to be able to define constraints but not have it checked > immediately. I envision it similar to how concurrent index creation > works: the constraint is created as "not checked" and the transaction is > committed; new insertions are checked against the constraint. Then the > table is scanned to verify that extant tuples pass the constraint, > _without_ the exclusive lock on the table. > > Both DB2 and Oracle have an ENFORCE setting for constraints, and a MySQL > blog hinted some time ago that it might be in SQL 201x. > > -- > Alvaro Herrera http://www.CommandPrompt.com/ > The PostgreSQL Company - Command Prompt, Inc. The mysql'ism foreign_key_checks would seem to do similar things...? (http://dev.mysql.com/doc/refman/5.1/en/server-session-variables.html#sysvar_foreign_key_checks ) -- Rob Wultsch wultsch@gmail.com
Alvaro Herrera <alvherre@commandprompt.com> wrote: > Both DB2 and Oracle have an ENFORCE setting for constraints, and a > MySQL blog hinted some time ago that it might be in SQL 201x. If I remember correctly, Sybase never checks the existing data when you add a constraint of any type (except for a unique constraint or primary key). That has occasionally been useful to me when a business rule has been identified which we want to enforce in an existing database, but there hasn't yet been enforcement of that rule. You can "plug the leak" first, then list the legacy problems and get those on a list for cleanup. If you insist that all preexisting bad data must be cleaned up before you can prevent more bad data from going in, you might never *get* clean because of a steady dribble of additional bad data while you are attempting cleanup. (Well, OK, you could always enforce the rule at some other layer and hope to get enough traction to correct the problems, but it is nice to have help from the DBMS in this regard, without having to code triggers to get there.) -Kevin
peter_e@gmx.net (Peter Eisentraut) writes: > On Tue, 2009-09-08 at 16:07 -0400, Alvaro Herrera wrote: >> Tom Lane wrote: >> > Michael Gould <mgould@intermodalsoftwaresolutions.net> writes: >> > > It would be nice if we could enable and disable column and table >> > > constraints. I believe that you can do this in Oracle but this is very >> > > handy for testing stored procedures and other external processes. >> > >> > Drop the constraint and re-add it later... >> >> That's not very useful when adding it later means grabbing an exclusive >> lock on the table for the whole duration of the full table scan required >> to check the table. > > It's also useful to define foreign keys for documentation purposes but > not enforce them for some reason. With the ALTER TABLE DISABLE TRIGGER functionality added in 8.3, we already have the ability to do this with foreign key constraints. That suggests a place for syntax to come from, I'd expect. -- let name="cbbrowne" and tld="ca.afilias.info" in name ^ "@" ^ tld;; Christopher Browne "Bother," said Pooh, "Eeyore, ready two photon torpedoes and lock phasers on the Heffalump, Piglet, meet me in transporter room three"
Christopher Browne <cbbrowne@ca.afilias.info> writes:
> With the ALTER TABLE DISABLE TRIGGER functionality added in 8.3, we
> already have the ability to do this with foreign key constraints.
That "feature" is a crock that should not be extended, because it
leaves it entirely on the user's shoulders whether the constraint
is actually true when the system thinks it is. What is being discussed
here is ways to incrementally add real, proven-valid constraints.
(Indeed, given the thought that's being given to having the planner
assume that FK constraints hold, I rather think that we need to
reconsider ALTER DISABLE TRIGGER.)
regards, tom lane
On 9/10/2009 11:06 AM, Tom Lane wrote: > Christopher Browne <cbbrowne@ca.afilias.info> writes: >> With the ALTER TABLE DISABLE TRIGGER functionality added in 8.3, we >> already have the ability to do this with foreign key constraints. > > That "feature" is a crock that should not be extended, because it > leaves it entirely on the user's shoulders whether the constraint > is actually true when the system thinks it is. What is being discussed > here is ways to incrementally add real, proven-valid constraints. > > (Indeed, given the thought that's being given to having the planner > assume that FK constraints hold, I rather think that we need to > reconsider ALTER DISABLE TRIGGER.) The feature was originally intended to be a clean way of avoiding interferences of triggers and/or foreign keys with replication systems that work on the user level (like Bucardo, Londiste and Slony). The only way to break foreign keys in that scenario is to replicate a referencing table without replicating the corresponding PK table. Note that Slony-I currently does apply updates in a fashion that would actually make checking of foreign keys on the replica possible, but does need the ability to disable regular user triggers. But for some future version of Slony, we may need to change that and apply changes within one replication group (SYNC) out of order with respect to multiple tables. Which means that Slony would need at least some mechanism to disable user triggers and force all foreign key constraints to be deferred, whether they are declared deferrable or not. Jan -- Anyone who trades liberty for security deserves neither liberty nor security. -- Benjamin Franklin
On Thu, Sep 10, 2009 at 04:24:15PM -0400, Jan Wieck wrote: > The feature was originally intended to be a clean way of avoiding > interferences of triggers and/or foreign keys with replication systems > that work on the user level (like Bucardo, Londiste and Slony). The only > way to break foreign keys in that scenario is to replicate a referencing > table without replicating the corresponding PK table. FWIW, I find the ability in Slony to configure triggers so they work or not depending on the replication role to be extremely useful. Absolutely a major positive feature. Have a nice day, -- Martijn van Oosterhout <kleptog@svana.org> http://svana.org/kleptog/ > Please line up in a tree and maintain the heap invariant while > boarding. Thank you for flying nlogn airlines.
>Martijn van Oosterhout <kleptog@svana.org> wrote: > FWIW, I find the ability in Slony to configure triggers so they work > or not depending on the replication role to be extremely useful. > Absolutely a major positive feature. Yeah, as a general rule it doesn't make sense to try to enforce constraints on a replication *target*. Check and report, perhaps, but you don't normally want to error out on anything which you know was actually applied to the source database. It's even worse for some classes of triggers which generate derived data; you don't want the replication to generate one value and then a trigger on the replication target to try to do the same. A count, for example, could easily wind up with an "off by one" error much of the time. -Kevin