Thread: Alpha releases: How to tag
As we are approaching the first alpha release, let's think about how to tag and label it and how to schedule those two operations with respect to one another. The typical process for, say, a beta release is - apply version number change to source tree - commit - tag (repeat for next beta) The problem, arguably, with this is that the source tree would then read 8.5alpha1 until alpha2, instead of the 8.5devel that we are used to. The fix could be: - apply version number change to source tree - commit - tag - revert version number change in source tree - commit (repeat for next alpha) or more sophisticatedly - branch - apply version number change to source tree - commit - tag or alternatively no tag at all, just apply version number and build tarball. Comments?
* Peter Eisentraut (peter_e@gmx.net) wrote:
> - branch
> - apply version number change to source tree
> - commit
> - tag
If this wasn't CVS, this would certainly be the way to go.
> or alternatively no tag at all, just apply version number and build tarball.
As this *is* CVS, I'm thinking we should probably go with this approach.
Just my 2c.
Thanks,
Stephen
Peter Eisentraut wrote: > As we are approaching the first alpha release, let's think about how to tag > and label it and how to schedule those two operations with respect to one > another. > > The typical process for, say, a beta release is > > - apply version number change to source tree > - commit > - tag > (repeat for next beta) > > The problem, arguably, with this is that the source tree would then read > 8.5alpha1 until alpha2, instead of the 8.5devel that we are used to. The fix > could be: > > - apply version number change to source tree > - commit > - tag > - revert version number change in source tree > - commit > (repeat for next alpha) > > or more sophisticatedly > > - branch > - apply version number change to source tree > - commit > - tag > > or alternatively no tag at all, just apply version number and build tarball. > > Comments? > > Does it need a version number change? Maybe just a tag (no branch) is all that is required. cheers andrew
On Aug 3, 2009, at 7:57 AM, Stephen Frost <sfrost@snowman.net> wrote: > * Peter Eisentraut (peter_e@gmx.net) wrote: >> - branch >> - apply version number change to source tree >> - commit >> - tag > > If this wasn't CVS, this would certainly be the way to go. > >> or alternatively no tag at all, just apply version number and build >> tarball. > > As this *is* CVS, I'm thinking we should probably go with this > approach. Sounds right to me. ...Robert
Andrew Dunstan <andrew@dunslane.net> writes:
> Does it need a version number change? Maybe just a tag (no branch) is
> all that is required.
I think that we do want the alpha releases to identify themselves as
such. And we want a marker in CVS as to what state the alpha release
corresponds to. Peter's label-and-undo approach seems like a kluge;
and it doesn't scale to consider the possibility that we might
want to re-release an alpha after fixing some particularly evil bug.
A tag without a branch won't handle that either.
I feel that making a branch is the way to go. If we try to get away
with a shortcut, we'll probably regret it.
regards, tom lane
Tom Lane wrote: > Andrew Dunstan <andrew@dunslane.net> writes: > >> Does it need a version number change? Maybe just a tag (no branch) is >> all that is required. >> > > I think that we do want the alpha releases to identify themselves as > such. And we want a marker in CVS as to what state the alpha release > corresponds to. Peter's label-and-undo approach seems like a kluge; > and it doesn't scale to consider the possibility that we might > want to re-release an alpha after fixing some particularly evil bug. > A tag without a branch won't handle that either. > > I feel that making a branch is the way to go. If we try to get away > with a shortcut, we'll probably regret it. > > > Yes, if that's what we want then definitely branch. I guess the branch will (almost) only ever have exactly one commit on it. cheers andrew
Andrew Dunstan <andrew@dunslane.net> writes:
> Tom Lane wrote:
>> ... it doesn't scale to consider the possibility that we might
>> want to re-release an alpha after fixing some particularly evil bug.
> Yes, if that's what we want then definitely branch. I guess the branch
> will (almost) only ever have exactly one commit on it.
Yeah, I'd expect so; but it seems like a good idea to leave room for the
possibility of more than one commit. I was first thinking that we'd
just release a new alpha from CVS HEAD if the evil-bug situation comes
up. However, if we've already done a catversion bump or some other
incompatible change in HEAD, that wouldn't be very practical.
regards, tom lane
On Monday 03 August 2009 17:44:32 Tom Lane wrote: > Andrew Dunstan <andrew@dunslane.net> writes: > > Does it need a version number change? Maybe just a tag (no branch) is > > all that is required. > > I think that we do want the alpha releases to identify themselves as > such. And we want a marker in CVS as to what state the alpha release > corresponds to. Peter's label-and-undo approach seems like a kluge; > and it doesn't scale to consider the possibility that we might > want to re-release an alpha after fixing some particularly evil bug. > A tag without a branch won't handle that either. > > I feel that making a branch is the way to go. If we try to get away > with a shortcut, we'll probably regret it. Another more lightweight alternative is to tag and then change the version number and build the tarball, without another commit. This assumes that the version number stamping is automated and reproducible, so you don't have to record it in history.
On Mon, Aug 03, 2009 at 10:44:32AM -0400, Tom Lane wrote: > Andrew Dunstan <andrew@dunslane.net> writes: > > Does it need a version number change? Maybe just a tag (no branch) > > is all that is required. > > I think that we do want the alpha releases to identify themselves as > such. And we want a marker in CVS as to what state the alpha > release corresponds to. Peter's label-and-undo approach seems like > a kluge; Right. > and it doesn't scale to consider the possibility that we might want > to re-release an alpha after fixing some particularly evil bug. A > tag without a branch won't handle that either. Is this a use case? I truly hope nobody will try using a beta, let alone an alpha, in production. Do we need to provide for such a possibility? I don't recall that we've ever back-patched a beta, or even a release candidate. Cheers, David. -- David Fetter <david@fetter.org> http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fetter@gmail.com Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate
David Fetter <david@fetter.org> writes:
> On Mon, Aug 03, 2009 at 10:44:32AM -0400, Tom Lane wrote:
>> and it doesn't scale to consider the possibility that we might want
>> to re-release an alpha after fixing some particularly evil bug. A
>> tag without a branch won't handle that either.
> Is this a use case? I truly hope nobody will try using a beta, let
> alone an alpha, in production. Do we need to provide for such a
> possibility? I don't recall that we've ever back-patched a beta, or
> even a release candidate.
I don't really know if it's a use-case or not; I just have a feeling
that if we use a release procedure that guarantees we can't do it,
we'll live to regret that.
The bug-fixing situation for betas and RCs is a bit different because
it's expected that there will be a compatible update available shortly.
So you can usually assume that updating to the next beta/RC/release will
fix whatever problems got found. Alphas are going to be out there on
their own with absolutely no expectation that the next alpha is
catversion-compatible. And I doubt we'd bother generating pg_migrator
builds that work for pairs of alpha releases.
I agree with you that it'd be insane to run anything mission-critical on
an alpha build; but I could see someone having loaded up quite a lot of
test data on one. (If we aren't hoping for some pretty serious testing
of these releases, I am not sure what is the point of doing them at all.)
So it seems to me that having the ability to fix show-stopper bugs
without forcing a migration to a later alpha would be a good thing.
Maybe we'll never need it, or maybe we will.
regards, tom lane
Peter Eisentraut <peter_e@gmx.net> writes:
> On Monday 03 August 2009 17:44:32 Tom Lane wrote:
>> I feel that making a branch is the way to go. If we try to get away
>> with a shortcut, we'll probably regret it.
> Another more lightweight alternative is to tag and then change the version
> number and build the tarball, without another commit. This assumes that the
> version number stamping is automated and reproducible, so you don't have to
> record it in history.
The stamping is reproducible enough, I think, but don't we have the
tarball build process set up so that it pulls the tarball directly
from a particular CVS tag?
Anyway, the reason I'm for a branch is not that. It's so we have the
option to issue an 8.5alpha1.1 update if we need to.
regards, tom lane
On Mon, Aug 03, 2009 at 11:32:48AM -0400, Tom Lane wrote: > David Fetter <david@fetter.org> writes: > > On Mon, Aug 03, 2009 at 10:44:32AM -0400, Tom Lane wrote: > >> and it doesn't scale to consider the possibility that we might want > >> to re-release an alpha after fixing some particularly evil bug. A > >> tag without a branch won't handle that either. > > > Is this a use case? I truly hope nobody will try using a beta, let > > alone an alpha, in production. Do we need to provide for such a > > possibility? I don't recall that we've ever back-patched a beta, or > > even a release candidate. > > I don't really know if it's a use-case or not; I just have a feeling > that if we use a release procedure that guarantees we can't do it, > we'll live to regret that. I can see being cautious. I'm just questioning the value of this precaution in particular. > The bug-fixing situation for betas and RCs is a bit different > because it's expected that there will be a compatible update > available shortly. So you can usually assume that updating to the > next beta/RC/release will fix whatever problems got found. Alphas > are going to be out there on their own with absolutely no > expectation that the next alpha is catversion-compatible. We've bumped catversion in beta and even RC, if I recall right. > And I doubt we'd bother generating pg_migrator builds that work for > pairs of alpha releases. That's an interesting idea. Shouldn't pg_migrator be mandated to work for *any* catversion bump? Cheers, David. -- David Fetter <david@fetter.org> http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fetter@gmail.com Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate
David Fetter <david@fetter.org> writes:
> On Mon, Aug 03, 2009 at 11:32:48AM -0400, Tom Lane wrote:
>> And I doubt we'd bother generating pg_migrator builds that work for
>> pairs of alpha releases.
> That's an interesting idea. Shouldn't pg_migrator be mandated to work
> for *any* catversion bump?
Oh, are you volunteering to make that happen? That code is going to be
ugly enough just dealing with pairs of major releases ...
regards, tom lane
On Mon, Aug 3, 2009 at 17:32, Tom Lane<tgl@sss.pgh.pa.us> wrote: > David Fetter <david@fetter.org> writes: >> On Mon, Aug 03, 2009 at 10:44:32AM -0400, Tom Lane wrote: >>> and it doesn't scale to consider the possibility that we might want >>> to re-release an alpha after fixing some particularly evil bug. A >>> tag without a branch won't handle that either. > >> Is this a use case? I truly hope nobody will try using a beta, let >> alone an alpha, in production. Do we need to provide for such a >> possibility? I don't recall that we've ever back-patched a beta, or >> even a release candidate. > > I don't really know if it's a use-case or not; I just have a feeling > that if we use a release procedure that guarantees we can't do it, > we'll live to regret that. Agreed. > The bug-fixing situation for betas and RCs is a bit different because > it's expected that there will be a compatible update available shortly. > So you can usually assume that updating to the next beta/RC/release will > fix whatever problems got found. Alphas are going to be out there on > their own with absolutely no expectation that the next alpha is > catversion-compatible. And I doubt we'd bother generating pg_migrator > builds that work for pairs of alpha releases. > > I agree with you that it'd be insane to run anything mission-critical on > an alpha build; but I could see someone having loaded up quite a lot of > test data on one. (If we aren't hoping for some pretty serious testing > of these releases, I am not sure what is the point of doing them at all.) > So it seems to me that having the ability to fix show-stopper bugs > without forcing a migration to a later alpha would be a good thing. > Maybe we'll never need it, or maybe we will. I think the alpha->alpha migration would actually be very useful to these testers. That'll get them onto the new alpha. I think that's actually a lot more important than alpha->release or release->next alpha. I haven't actually looked into pg_migrator enough to know how likely it is that it'll "just work" going alpha->alpha when there have only been "normal" changes? How invasive are the changes that actually require pg_migrator to be touched at all? -- Magnus HaganderSelf: http://www.hagander.net/Work: http://www.redpill-linpro.com/
Magnus Hagander <magnus@hagander.net> writes:
> I haven't actually looked into pg_migrator enough to know how likely
> it is that it'll "just work" going alpha->alpha when there have only
> been "normal" changes? How invasive are the changes that actually
> require pg_migrator to be touched at all?
To my mind there are three categories of stuff that pg_migrator has to
do:
* catalog changes (handled by pg_dump/reload)* on-disk data layout changes (not handled yet anyway)* random weird
unclassifiablestuff
It's the third category that is the problem. An example from the 8.3 to
8.4 migration is the need to specially handle sequences because they're
not compatible on-disk. Any pair of releases you might pick is going to
have its own little quirks like that. Our intention (at least as I see
it) is to minimize pg_migrator's complexity by decreeing that any given
release of pg_migrator deals with exactly one pair of PG releases.
I don't think that works if alpha->alpha has to be supported --- in
particular, 8.5alpha1 to 8.5alpha2 might be quite different from what
will be needed later for 8.4 to 8.5, so you would end up with multiple
live branches of pg_migrator, or else a lot of spaghetti code trying to
track which actions to take for a given case.
The other little problem is who's going to do this work and when.
The alpha-release idea was sold to us on the basis of being a small
amount of incremental work: just tag an alpha release right after
CommitFest and kick it out there. If we're waiting around for someone
to produce and test a compatible pg_migrator, that's not going to be
how it works.
regards, tom lane
On Mon, Aug 03, 2009 at 12:19:40PM -0400, Tom Lane wrote: > David Fetter <david@fetter.org> writes: > > On Mon, Aug 03, 2009 at 11:32:48AM -0400, Tom Lane wrote: > >> And I doubt we'd bother generating pg_migrator builds that work > >> for pairs of alpha releases. > > > That's an interesting idea. Shouldn't pg_migrator be mandated to > > work for *any* catversion bump? > > Oh, are you volunteering to make that happen? That code is going to > be ugly enough just dealing with pairs of major releases ... With all due respect, if pg_migrator does not function in such a way as to have data-driven composable changes, it's going to bang us up in the long haul much worse than not branching alphas ever could. We require that people supply docs with their changes, and it is totally unreasonable to let them send in catalog changes which do not include need migration changes. That's how it works in every other RDBMS outfit that has changes on disk, and we do not need to be the exception. If pg_migrator doesn't have this ability, we need to refactor it until it does, or go with something that can. Cheers, David. -- David Fetter <david@fetter.org> http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fetter@gmail.com Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate
On Monday 03 August 2009 21:07:00 David Fetter wrote: > We require that people supply docs with their changes, and it is > totally unreasonable to let them send in catalog changes which do not > include need migration changes. That's how it works in every other > RDBMS outfit that has changes on disk, and we do not need to be the > exception. Well, blocker number one for that is that pg_migrator is not even in the PostgreSQL CVS repository, but is more like an endorsed third-party product.
On Mon, Aug 3, 2009 at 2:07 PM, David Fetter<david@fetter.org> wrote: > On Mon, Aug 03, 2009 at 12:19:40PM -0400, Tom Lane wrote: >> David Fetter <david@fetter.org> writes: >> > On Mon, Aug 03, 2009 at 11:32:48AM -0400, Tom Lane wrote: >> >> And I doubt we'd bother generating pg_migrator builds that work >> >> for pairs of alpha releases. >> >> > That's an interesting idea. Shouldn't pg_migrator be mandated to >> > work for *any* catversion bump? >> >> Oh, are you volunteering to make that happen? That code is going to >> be ugly enough just dealing with pairs of major releases ... > > With all due respect, if pg_migrator does not function in such a way > as to have data-driven composable changes, it's going to bang us up in > the long haul much worse than not branching alphas ever could. > > We require that people supply docs with their changes, and it is > totally unreasonable to let them send in catalog changes which do not > include need migration changes. That's how it works in every other > RDBMS outfit that has changes on disk, and we do not need to be the > exception. > > If pg_migrator doesn't have this ability, we need to refactor it until > it does, or go with something that can. I don't disagree with this, but we don't have anyone to do the work atm, even if it were exactly clear what to do, which it isn't. What we could do, though, is try to figure out whether it will work between 8.4.0 and 8.5alpha1. ...Robert
On Mon, Aug 03, 2009 at 09:22:52PM +0300, Peter Eisentraut wrote: > On Monday 03 August 2009 21:07:00 David Fetter wrote: > > We require that people supply docs with their changes, and it is > > totally unreasonable to let them send in catalog changes which do > > not include need migration changes. That's how it works in every > > other RDBMS outfit that has changes on disk, and we do not need to > > be the exception. > > Well, blocker number one for that is that pg_migrator is not even in > the PostgreSQL CVS repository, but is more like an endorsed > third-party product. I'm not entirely sure that pg_migrator should be tied to releases of PostgreSQL, given what it does. Or did you mean that it's not been given the same scrutiny that the PostgreSQL code base has? Thoughts? Cheers, David. -- David Fetter <david@fetter.org> http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fetter@gmail.com Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate
On Monday 03 August 2009 22:52:55 David Fetter wrote: > On Mon, Aug 03, 2009 at 09:22:52PM +0300, Peter Eisentraut wrote: > > On Monday 03 August 2009 21:07:00 David Fetter wrote: > > > We require that people supply docs with their changes, and it is > > > totally unreasonable to let them send in catalog changes which do > > > not include need migration changes. That's how it works in every > > > other RDBMS outfit that has changes on disk, and we do not need to > > > be the exception. > > > > Well, blocker number one for that is that pg_migrator is not even in > > the PostgreSQL CVS repository, but is more like an endorsed > > third-party product. > > I'm not entirely sure that pg_migrator should be tied to releases of > PostgreSQL, given what it does. Or did you mean that it's not been > given the same scrutiny that the PostgreSQL code base has? Well, how to you expect to mandate that all patches come with upgrade support if the PostgreSQL software proper does not contain any upgrade support in the first place, because pg_migrator is maintained elsewhere.
On Tue, Aug 04, 2009 at 05:19:16PM +0300, Peter Eisentraut wrote: > On Monday 03 August 2009 22:52:55 David Fetter wrote: > > On Mon, Aug 03, 2009 at 09:22:52PM +0300, Peter Eisentraut wrote: > > > On Monday 03 August 2009 21:07:00 David Fetter wrote: > > > > We require that people supply docs with their changes, and it > > > > is totally unreasonable to let them send in catalog changes > > > > which do not include need migration changes. That's how it > > > > works in every other RDBMS outfit that has changes on disk, > > > > and we do not need to be the exception. > > > > > > Well, blocker number one for that is that pg_migrator is not > > > even in the PostgreSQL CVS repository, but is more like an > > > endorsed third-party product. > > > > I'm not entirely sure that pg_migrator should be tied to releases > > of PostgreSQL, given what it does. Or did you mean that it's not > > been given the same scrutiny that the PostgreSQL code base has? > > Well, how to you expect to mandate that all patches come with > upgrade support if the PostgreSQL software proper does not contain > any upgrade support in the first place, because pg_migrator is > maintained elsewhere. If I'm understanding you correctly, you're saying that pg_migrator (or whatever actually does this) needs to be an official PostgreSQL project in order for us to be able to require that people use it. For what it's worth, I agree. Is it strictly necessary that its release cycles match exactly those of the database engine, or would it be OK for it to release as needed, not triggering a major PostgreSQL release? Cheers, David. -- David Fetter <david@fetter.org> http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fetter@gmail.com Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate
David Fetter wrote: > Is it strictly necessary that its release cycles match exactly those > of the database engine, or would it be OK for it to release as needed, > not triggering a major PostgreSQL release? Well, pg_migrator already released an 8.4.1 ... -- Alvaro Herrera http://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc.
On Wednesday 05 August 2009 06:00:19 David Fetter wrote: > If I'm understanding you correctly, you're saying that pg_migrator (or > whatever actually does this) needs to be an official PostgreSQL > project in order for us to be able to require that people use it. For > what it's worth, I agree. > > Is it strictly necessary that its release cycles match exactly those > of the database engine, or would it be OK for it to release as needed, > not triggering a major PostgreSQL release? Right now, anything is possible. It mainly needs people to make something happen. The current maintainers of pg_migrator are still focused on making the 8.3 -> 8.4 path working robustly. And after that, they will likely take a long rest. If people want pg_migrator to stay current with 8.5devel, they need to take it upon themselves to create repeatable tests and code up the necessary changes. And then later if that turns out to be a viable undertaking, we can consider whether we merge pg_migrator and make updating it a requirement for any patch.
Peter Eisentraut wrote: > On Monday 03 August 2009 21:07:00 David Fetter wrote: > > We require that people supply docs with their changes, and it is > > totally unreasonable to let them send in catalog changes which do not > > include need migration changes. That's how it works in every other > > RDBMS outfit that has changes on disk, and we do not need to be the > > exception. > > Well, blocker number one for that is that pg_migrator is not even in the > PostgreSQL CVS repository, but is more like an endorsed third-party product. I wouldn't say pg_migrator is "endorsed". It is on pgfoundry and was mentioned in the press release, but it isn't mentioned in our documentation about upgrading, it wasn't mentioned in the release notes, and it isn't mentioned on our web site, except as a news item. I believe this is because of concerns about pg_migrator's "experimental" nature. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. +
On Fri, Aug 7, 2009 at 11:35 AM, Bruce Momjian<bruce@momjian.us> wrote: > Peter Eisentraut wrote: >> On Monday 03 August 2009 21:07:00 David Fetter wrote: >> > We require that people supply docs with their changes, and it is >> > totally unreasonable to let them send in catalog changes which do not >> > include need migration changes. That's how it works in every other >> > RDBMS outfit that has changes on disk, and we do not need to be the >> > exception. >> >> Well, blocker number one for that is that pg_migrator is not even in the >> PostgreSQL CVS repository, but is more like an endorsed third-party product. > > I wouldn't say pg_migrator is "endorsed". It is on pgfoundry and was > mentioned in the press release, but it isn't mentioned in our > documentation about upgrading, it wasn't mentioned in the release notes, > and it isn't mentioned on our web site, except as a news item. > > I believe this is because of concerns about pg_migrator's "experimental" > nature. I think so. And also because it has a fair number of documented restrictions. Hopefully we'll be able to remove some of those in a future release. ...Robert
Robert Haas wrote: > On Mon, Aug 3, 2009 at 2:07 PM, David Fetter<david@fetter.org> wrote: > > On Mon, Aug 03, 2009 at 12:19:40PM -0400, Tom Lane wrote: > >> David Fetter <david@fetter.org> writes: > >> > On Mon, Aug 03, 2009 at 11:32:48AM -0400, Tom Lane wrote: > >> >> And I doubt we'd bother generating pg_migrator builds that work > >> >> for pairs of alpha releases. > >> > >> > That's an interesting idea. ?Shouldn't pg_migrator be mandated to > >> > work for *any* catversion bump? > >> > >> Oh, are you volunteering to make that happen? ?That code is going to > >> be ugly enough just dealing with pairs of major releases ... > > > > With all due respect, if pg_migrator does not function in such a way > > as to have data-driven composable changes, it's going to bang us up in > > the long haul much worse than not branching alphas ever could. > > > > We require that people supply docs with their changes, and it is > > totally unreasonable to let them send in catalog changes which do not > > include need migration changes. ?That's how it works in every other > > RDBMS outfit that has changes on disk, and we do not need to be the > > exception. > > > > If pg_migrator doesn't have this ability, we need to refactor it until > > it does, or go with something that can. > > I don't disagree with this, but we don't have anyone to do the work > atm, even if it were exactly clear what to do, which it isn't. What > we could do, though, is try to figure out whether it will work between > 8.4.0 and 8.5alpha1. I just tested this and it works fine. In fact it even does 8.3 to 8.5devel, with no pg_migrator changes needed. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. +
Alvaro Herrera wrote: > David Fetter wrote: > > > Is it strictly necessary that its release cycles match exactly those > > of the database engine, or would it be OK for it to release as needed, > > not triggering a major PostgreSQL release? > > Well, pg_migrator already released an 8.4.1 ... Well, actually at 8.4.4 right now, but who's counting. It is clear that pg_migrator has to be separate so it can rapidly address deficiencies without being tied to database server releases. We found a problem with arrays/composites/enums two days ago and had it fixed/detected in 24 hours. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. +
Magnus Hagander wrote: > > The bug-fixing situation for betas and RCs is a bit different because > > it's expected that there will be a compatible update available shortly. > > So you can usually assume that updating to the next beta/RC/release will > > fix whatever problems got found. ?Alphas are going to be out there on > > their own with absolutely no expectation that the next alpha is > > catversion-compatible. ?And I doubt we'd bother generating pg_migrator > > builds that work for pairs of alpha releases. > > > > I agree with you that it'd be insane to run anything mission-critical on > > an alpha build; but I could see someone having loaded up quite a lot of > > test data on one. ?(If we aren't hoping for some pretty serious testing > > of these releases, I am not sure what is the point of doing them at all.) > > So it seems to me that having the ability to fix show-stopper bugs > > without forcing a migration to a later alpha would be a good thing. > > Maybe we'll never need it, or maybe we will. > > I think the alpha->alpha migration would actually be very useful to > these testers. That'll get them onto the new alpha. I think that's > actually a lot more important than alpha->release or release->next > alpha. > > I haven't actually looked into pg_migrator enough to know how likely > it is that it'll "just work" going alpha->alpha when there have only > been "normal" changes? How invasive are the changes that actually > require pg_migrator to be touched at all? pg_migrator handles 8.5devel just fine now with no changes. What will happen is that people will try pg_migrator in 8.5devel and if it fails we will hear about it and and adjust pg_migrator (or the backend) to get it working. This will give us early warning about backend changes and hopefully make pg_migrator more useful when 8.5 final is released. What I have found is that pg_migrator has a useful toolkit for detecting and correcting database incompatibilities. It wasn't easy to get working but now that it is complete adding stuff is actually easy --- see the enum/array/composite fix that was done yesterday within 24 hours of the report. Now, I am not guaranteeing that pg_migrator is always going to work for 8.5devel, but while it does let's use it, and when it fails, we can just update the README to say it doesn't work for 8.5devel, or fix things. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. +
Tom Lane wrote: > Magnus Hagander <magnus@hagander.net> writes: > > I haven't actually looked into pg_migrator enough to know how likely > > it is that it'll "just work" going alpha->alpha when there have only > > been "normal" changes? How invasive are the changes that actually > > require pg_migrator to be touched at all? > > To my mind there are three categories of stuff that pg_migrator has to > do: > > * catalog changes (handled by pg_dump/reload) > * on-disk data layout changes (not handled yet anyway) > * random weird unclassifiable stuff > > It's the third category that is the problem. An example from the 8.3 to > 8.4 migration is the need to specially handle sequences because they're > not compatible on-disk. Any pair of releases you might pick is going to > have its own little quirks like that. Our intention (at least as I see > it) is to minimize pg_migrator's complexity by decreeing that any given > release of pg_migrator deals with exactly one pair of PG releases. > I don't think that works if alpha->alpha has to be supported --- in > particular, 8.5alpha1 to 8.5alpha2 might be quite different from what > will be needed later for 8.4 to 8.5, so you would end up with multiple > live branches of pg_migrator, or else a lot of spaghetti code trying to > track which actions to take for a given case. > > The other little problem is who's going to do this work and when. > The alpha-release idea was sold to us on the basis of being a small > amount of incremental work: just tag an alpha release right after > CommitFest and kick it out there. If we're waiting around for someone > to produce and test a compatible pg_migrator, that's not going to be > how it works. Because the 8.5 is alpha anyway, and because pg_migrator works with current CVS, let's just say it works and wait for someone to report a problem. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. +
David Fetter wrote: > On Mon, Aug 03, 2009 at 12:19:40PM -0400, Tom Lane wrote: > > David Fetter <david@fetter.org> writes: > > > On Mon, Aug 03, 2009 at 11:32:48AM -0400, Tom Lane wrote: > > >> And I doubt we'd bother generating pg_migrator builds that work > > >> for pairs of alpha releases. > > > > > That's an interesting idea. Shouldn't pg_migrator be mandated to > > > work for *any* catversion bump? > > > > Oh, are you volunteering to make that happen? That code is going to > > be ugly enough just dealing with pairs of major releases ... > > With all due respect, if pg_migrator does not function in such a way > as to have data-driven composable changes, it's going to bang us up in > the long haul much worse than not branching alphas ever could. > > We require that people supply docs with their changes, and it is > totally unreasonable to let them send in catalog changes which do not > include need migration changes. That's how it works in every other > RDBMS outfit that has changes on disk, and we do not need to be the > exception. > > If pg_migrator doesn't have this ability, we need to refactor it until > it does, or go with something that can. Odds are that the patch submitters will not understand enough to know how to modify pg_migrator, but just knowing something broke is usually enough for the hackers group to find a fix. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. +
On Fri, Aug 07, 2009 at 04:07:08PM -0400, Bruce Momjian wrote: > David Fetter wrote: > > On Mon, Aug 03, 2009 at 12:19:40PM -0400, Tom Lane wrote: > > > David Fetter <david@fetter.org> writes: > > > > On Mon, Aug 03, 2009 at 11:32:48AM -0400, Tom Lane wrote: > > > >> And I doubt we'd bother generating pg_migrator builds that > > > >> work for pairs of alpha releases. > > > > > > > That's an interesting idea. Shouldn't pg_migrator be mandated > > > > to work for *any* catversion bump? > > > > > > Oh, are you volunteering to make that happen? That code is > > > going to be ugly enough just dealing with pairs of major > > > releases ... > > > > With all due respect, if pg_migrator does not function in such a > > way as to have data-driven composable changes, it's going to bang > > us up in the long haul much worse than not branching alphas ever > > could. > > > > We require that people supply docs with their changes, and it is > > totally unreasonable to let them send in catalog changes which do > > not include need migration changes. That's how it works in every > > other RDBMS outfit that has changes on disk, and we do not need to > > be the exception. > > > > If pg_migrator doesn't have this ability, we need to refactor it > > until it does, or go with something that can. > > Odds are that the patch submitters will not understand enough to > know how to modify pg_migrator, but just knowing something broke is > usually enough for the hackers group to find a fix. This is a pretty serious drawback. If we're going to require that people send migration scripts when they change the on-disk format, this needs to be easy. The program, whatever it turns out to be, needs to take composable changes which people would submit along with their patch, just as they do docs. How much refactoring of pg_migrator would such a change take? Would it be worthwhile to do that, or just start a different project? Cheers, David. -- David Fetter <david@fetter.org> http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fetter@gmail.com Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate
David Fetter wrote: > This is a pretty serious drawback. If we're going to require that > people send migration scripts when they change the on-disk format, > this needs to be easy. But, are we? -- Alvaro Herrera http://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc.
On Fri, Aug 07, 2009 at 05:32:13PM -0400, Alvaro Herrera wrote: > David Fetter wrote: > > > This is a pretty serious drawback. If we're going to require that > > people send migration scripts when they change the on-disk format, > > this needs to be easy. > > But, are we? This is an area where we have been sorely lacking and sloppy for years. Every other RDBMS development team requires it. Cheers, David. -- David Fetter <david@fetter.org> http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fetter@gmail.com Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate
Alvaro Herrera wrote: > David Fetter wrote: > > >> This is a pretty serious drawback. If we're going to require that >> people send migration scripts when they change the on-disk format, >> this needs to be easy. >> > > But, are we? > > I think it's a lot more nebulous than that. At the same time I think the days when we can blithely change the on-disk format with hardly a thought for migration are over. IOW, there's agreement things have to change, but the exact shape of the change is not yet clear (at least to me) ;-) cheers andrew
Andrew Dunstan <andrew@dunslane.net> writes:
> I think it's a lot more nebulous than that. At the same time I think the
> days when we can blithely change the on-disk format with hardly a
> thought for migration are over. IOW, there's agreement things have to
> change, but the exact shape of the change is not yet clear (at least to
> me) ;-)
Yeah. I think we're going to start paying more than zero attention to
this, but we don't yet have a handle on what the real parameters are.
In particular, it's hard to argue that pg_migrator has yet achieved
more than experimental status, so accepting or rejecting patches on
the grounds of whether they would or would not break pg_migrator might
be a bit premature. And at the other end of the spectrum, nobody except
Zdenek wants to deal with changes as invasive as the ones he's proposed.
So we're still feeling our way here. We do *not* have a framework in
which someone could submit a patch that includes an on-disk migration
aspect, so David's position that we should immediately institute a
hard requirement for such seems a bit ivory-tower. We might as well
just say that the on-disk format is frozen, because that's what the
effect would be.
But having said all that, I'm okay with a go-slow position on on-disk
changes for the next little while. That would give us some breathing
room to see if something like pg_migrator is really workable at all.
We need to find out just how far that approach goes before we can make
many decisions in this area.
regards, tom lane
On Fri, Aug 07, 2009 at 06:02:32PM -0400, Tom Lane wrote: > Andrew Dunstan <andrew@dunslane.net> writes: > > I think it's a lot more nebulous than that. At the same time I think the > > days when we can blithely change the on-disk format with hardly a > > thought for migration are over. IOW, there's agreement things have to > > change, but the exact shape of the change is not yet clear (at least to > > me) ;-) > > Yeah. I think we're going to start paying more than zero attention to > this, but we don't yet have a handle on what the real parameters are. > In particular, it's hard to argue that pg_migrator has yet achieved > more than experimental status, so accepting or rejecting patches on > the grounds of whether they would or would not break pg_migrator might > be a bit premature. And at the other end of the spectrum, nobody except > Zdenek wants to deal with changes as invasive as the ones he's proposed. > So we're still feeling our way here. We do *not* have a framework in > which someone could submit a patch that includes an on-disk migration > aspect, so David's position that we should immediately institute a > hard requirement for such seems a bit ivory-tower. I am not suggesting that this change be immediate, and it's not ivory tower. It's just how everybody else does it. Cheers, David. -- David Fetter <david@fetter.org> http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fetter@gmail.com Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate
David Fetter <david@fetter.org> writes:
> I am not suggesting that this change be immediate, and it's not ivory
> tower. It's just how everybody else does it.
You keep saying that, and it's completely meaningless. What do you know
about the development practices of Oracle, or DB2, or even Mysql?
regards, tom lane
Tom Lane wrote: > David Fetter <david@fetter.org> writes: > >> I am not suggesting that this change be immediate, and it's not ivory >> tower. It's just how everybody else does it. >> > > You keep saying that, and it's completely meaningless. What do you know > about the development practices of Oracle, or DB2, or even Mysql? > > > The other point is that we don't change the on-disk format for fun, but for good reason. If it's going to get harder we'll pay a price. That doesn't mean we should go on as we have been, but just saying "we've been sloppy" is telling less than half the story. If one believes the rumors, Oracle at least has in the past paid dearly for supporting old formats. cheers andrew
David Fetter wrote: > > Odds are that the patch submitters will not understand enough to > > know how to modify pg_migrator, but just knowing something broke is > > usually enough for the hackers group to find a fix. > > This is a pretty serious drawback. If we're going to require that > people send migration scripts when they change the on-disk format, > this needs to be easy. The program, whatever it turns out to be, > needs to take composable changes which people would submit along with > their patch, just as they do docs. > > How much refactoring of pg_migrator would such a change take? Would > it be worthwhile to do that, or just start a different project? I have no idea what you are thinking. pg_migrator is in C just like the backend code. Do you want some plugin module to do migrations? I have neither the time or desire to implement that. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. +
On Aug 7, 2009, at 11:50 PM, Bruce Momjian <bruce@momjian.us> wrote: > David Fetter wrote: >>> Odds are that the patch submitters will not understand enough to >>> know how to modify pg_migrator, but just knowing something broke is >>> usually enough for the hackers group to find a fix. >> >> This is a pretty serious drawback. If we're going to require that >> people send migration scripts when they change the on-disk format, >> this needs to be easy. The program, whatever it turns out to be, >> needs to take composable changes which people would submit along with >> their patch, just as they do docs. >> >> How much refactoring of pg_migrator would such a change take? Would >> it be worthwhile to do that, or just start a different project? > > I have no idea what you are thinking. pg_migrator is in C just like > the > backend code. Do you want some plugin module to do migrations? I > have > neither the time or desire to implement that. It seems that it would also negate one of the major benefits of pg_migrator, which is that it doesn't require you to rewrite all of your data. ...Robert
Robert Haas wrote: > On Aug 7, 2009, at 11:50 PM, Bruce Momjian <bruce@momjian.us> wrote: > > > David Fetter wrote: > >>> Odds are that the patch submitters will not understand enough to > >>> know how to modify pg_migrator, but just knowing something broke is > >>> usually enough for the hackers group to find a fix. > >> > >> This is a pretty serious drawback. If we're going to require that > >> people send migration scripts when they change the on-disk format, > >> this needs to be easy. The program, whatever it turns out to be, > >> needs to take composable changes which people would submit along with > >> their patch, just as they do docs. > >> > >> How much refactoring of pg_migrator would such a change take? Would > >> it be worthwhile to do that, or just start a different project? > > > > I have no idea what you are thinking. pg_migrator is in C just like > > the > > backend code. Do you want some plugin module to do migrations? I > > have > > neither the time or desire to implement that. > > It seems that it would also negate one of the major benefits of > pg_migrator, which is that it doesn't require you to rewrite all of > your data. I don't understand how this relates to rewriting data. David was asking if we should require patch submitters to update pg_migrator. If they can do that, great, but if not, we can figure out an approach for them. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. +
On Fri, Aug 07, 2009 at 06:28:34PM -0400, Tom Lane wrote: > David Fetter <david@fetter.org> writes: > > I am not suggesting that this change be immediate, and it's not ivory > > tower. It's just how everybody else does it. > > You keep saying that, and it's completely meaningless. What do you know > about the development practices of Oracle, or DB2, or even Mysql? When I was at Sybase, changes to the on disk structure were required to provide code to do the migration. Nonetheless, at release time, the migrate process was almost always discovered to be broken, sometimes even before it was shipped to customers. Of course, Sybase implemented its own complete filesystem layer on top of raw partitions, so there was more scope to go wrong, especially since it was possible to corrupt the on disk structure in subtle ways that would not be discovered in normal operation but that would cause migration to corrupt it still further. In fairness, this is a very difficult problem to solve well and I expect to rely on dump/load migrations for quite sometime. -dg -- David Gould daveg@sonic.net 510 536 1443 510 282 0869 If simplicity worked, the world would be overrun with insects.
On Saturday 08 August 2009 01:28:34 Tom Lane wrote: > David Fetter <david@fetter.org> writes: > > I am not suggesting that this change be immediate, and it's not ivory > > tower. It's just how everybody else does it. > > You keep saying that, and it's completely meaningless. What do you know > about the development practices of Oracle, or DB2, or even Mysql? Whenever MySQL requires a disk format change, they write a new storage engine.
Tom Lane wrote: > Andrew Dunstan <andrew@dunslane.net> writes: > > I think it's a lot more nebulous than that. At the same time I think the > > days when we can blithely change the on-disk format with hardly a > > thought for migration are over. IOW, there's agreement things have to > > change, but the exact shape of the change is not yet clear (at least to > > me) ;-) > > Yeah. I think we're going to start paying more than zero attention to > this, but we don't yet have a handle on what the real parameters are. > In particular, it's hard to argue that pg_migrator has yet achieved > more than experimental status, so accepting or rejecting patches on > the grounds of whether they would or would not break pg_migrator might > be a bit premature. And at the other end of the spectrum, nobody except > Zdenek wants to deal with changes as invasive as the ones he's proposed. > So we're still feeling our way here. We do *not* have a framework in > which someone could submit a patch that includes an on-disk migration > aspect, so David's position that we should immediately institute a > hard requirement for such seems a bit ivory-tower. We might as well > just say that the on-disk format is frozen, because that's what the > effect would be. > > But having said all that, I'm okay with a go-slow position on on-disk > changes for the next little while. That would give us some breathing > room to see if something like pg_migrator is really workable at all. > We need to find out just how far that approach goes before we can make > many decisions in this area. I would be happy if we can just allow pg_migrator to _detect_ incompatible changes, and that nothing _major_ changes between releases that pg_migrator can't fix (we fixed the sequence changes in 8.4). /contrib changes are part of the first group, so I am happy to know about them and detect them; changing the tuple format, on the other hand, could make pg_migrator useless. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. +
On Monday 03 August 2009 17:44:32 Tom Lane wrote: > Andrew Dunstan <andrew@dunslane.net> writes: > > Does it need a version number change? Maybe just a tag (no branch) is > > all that is required. > > I think that we do want the alpha releases to identify themselves as > such. And we want a marker in CVS as to what state the alpha release > corresponds to. Peter's label-and-undo approach seems like a kluge; > and it doesn't scale to consider the possibility that we might > want to re-release an alpha after fixing some particularly evil bug. > A tag without a branch won't handle that either. > > I feel that making a branch is the way to go. If we try to get away > with a shortcut, we'll probably regret it. Well then, naming: Branch: ??? Tag: REL8_5_ALPHA1
daveg <daveg@sonic.net> wrote: > When I was at Sybase, changes to the on disk structure were required > to provide code to do the migration. Nonetheless, at release time, > the migrate process was almost always discovered to be broken, > sometimes even before it was shipped to customers. As a long-time user of Sybase SQL Server and it's later incarnation as Adaptive Server Enterprise, I can confirm that the result of this approach was often unsatisfactory from a user perspective. That is, the discovery was not always made before shipping to customers. :-( I have high hopes for pg_migrator, but the pg_dump | psql approach has always worked well for us -- no surprises and no pain beyond the resource management issues. -Kevin