Thread: Planning incompatibilities for Postgres 10.0
There are a number of changes we'd probably like to make to the way things work in Postgres. This thread is not about discussing what those are, just to say that requirements exist and have been discussed in various threads over time. The constraint on such changes is that we've decided that we must have an upgrade path from release to release. So I'd like to make a formal suggestion of a plan for how we cope with this: 1. Implement online upgrade in 9.4 via the various facilities we have in-progress. That looks completely possible. 2. Name the next release after that 10.0 (would have been 9.5). We declare now that a) 10.0 will support on-line upgrade from 9.4 (only) b) various major incompatibilities will be introduced in 10.0 - the change in release number will indicate to everybody that is the case c) agree that there will be no pg_upgrade patch from 9.4 to 10.0, so that we will not be constrained by that This plan doesn't presume any particular change. Each change would need to be discussed on a separate thread, with a separate case for each. All I'm suggesting is that we have a coherent plan for the timing of such changes, so we can bundle them together into one release. By doing this now we give ourselves lots of time to plan changes that will see us good for another decade. If we don't do this, then we simply risk losing the iniative by continuing to support legacy formats and approaches. --Simon Riggs http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
On Sat, May 25, 2013 at 4:39 AM, Simon Riggs <simon@2ndquadrant.com> wrote: > There are a number of changes we'd probably like to make to the way > things work in Postgres. This thread is not about discussing what > those are, just to say that requirements exist and have been discussed > in various threads over time. > > The constraint on such changes is that we've decided that we must have > an upgrade path from release to release. > > So I'd like to make a formal suggestion of a plan for how we cope with this: > > 1. Implement online upgrade in 9.4 via the various facilities we have > in-progress. That looks completely possible. > > 2. Name the next release after that 10.0 (would have been 9.5). We > declare now that > a) 10.0 will support on-line upgrade from 9.4 (only) > b) various major incompatibilities will be introduced in 10.0 - the > change in release number will indicate to everybody that is the case > c) agree that there will be no pg_upgrade patch from 9.4 to 10.0, so > that we will not be constrained by that > > This plan doesn't presume any particular change. Each change would > need to be discussed on a separate thread, with a separate case for > each. All I'm suggesting is that we have a coherent plan for the > timing of such changes, so we can bundle them together into one > release. > > By doing this now we give ourselves lots of time to plan changes that > will see us good for another decade. If we don't do this, then we > simply risk losing the iniative by continuing to support legacy > formats and approaches. Huh. I don't think that bumping the version number to 10.0 vs 9.5 is justification to introduce breaking changes. In fact, I would rather see 10.0 be the version where we formally stop doing that. I understand that some stuff needs to be improved but it often doesn't seem to be worth the cost in the long run. merlin
On Sat, 2013-05-25 at 10:39 +0100, Simon Riggs wrote: > The constraint on such changes is that we've decided that we must have > an upgrade path from release to release. Is this proposal only relaxing the binary upgrade requirement, or would it also relax other compatibility requirements, such as language and API compatibility? We need a couple major drivers of the incompatibility that really show users some value for going through the upgrade pain. Preferably, at least one would be a serious performance boost, because the users that encounter the most logical upgrade pain are also the ones that need a performance boost the most. Before we set a specific schedule, I think it would be a good idea to start prototyping some performance improvements that involve breaking the data format. Then, depending on how achievable it is, we can plan for however many more 9.X releases we think we need. That being said, I agree with you that planning in advance is important here, so that everyone knows when they need to get format-breaking changes in by. Regards,Jeff Davis
On 25 May 2013 18:13, Jeff Davis <pgsql@j-davis.com> wrote: > On Sat, 2013-05-25 at 10:39 +0100, Simon Riggs wrote: >> The constraint on such changes is that we've decided that we must have >> an upgrade path from release to release. > > Is this proposal only relaxing the binary upgrade requirement, or would > it also relax other compatibility requirements, such as language and API > compatibility? I'm suggesting that as many as possible changes we would like to make can happen in one release. This is for the benefit of users, so we dont make every release a source of incompatibilities. And that release should be the first one where we have online upgrade possible, which is one after next. --Simon Riggs http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
On Sat, May 25, 2013 at 10:39:30AM +0100, Simon Riggs wrote: > There are a number of changes we'd probably like to make to the way > things work in Postgres. This thread is not about discussing what > those are, just to say that requirements exist and have been discussed > in various threads over time. > > The constraint on such changes is that we've decided that we must have > an upgrade path from release to release. > > So I'd like to make a formal suggestion of a plan for how we cope with this: > > 1. Implement online upgrade in 9.4 via the various facilities we have > in-progress. That looks completely possible. > > 2. Name the next release after that 10.0 (would have been 9.5). We > declare now that > a) 10.0 will support on-line upgrade from 9.4 (only) > b) various major incompatibilities will be introduced in 10.0 - the > change in release number will indicate to everybody that is the case > c) agree that there will be no pg_upgrade patch from 9.4 to 10.0, so > that we will not be constrained by that Assuming online upgrade is going to require logical replication, you are also assuming 2x storage as you need to have a second cluster to perform the upgrade. pg_upgrade would still be needed to upgrade a cluster in-place. This sounds like, "I created a new tool which does some of what the old tool does. Let's break the old tool to allow some unspecified changes I might want to make." I consider this thread to be not thought-through, obviously. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. +
On 25 May 2013 21:44, Bruce Momjian <bruce@momjian.us> wrote: > On Sat, May 25, 2013 at 10:39:30AM +0100, Simon Riggs wrote: >> There are a number of changes we'd probably like to make to the way >> things work in Postgres. This thread is not about discussing what >> those are, just to say that requirements exist and have been discussed >> in various threads over time. >> >> The constraint on such changes is that we've decided that we must have >> an upgrade path from release to release. >> >> So I'd like to make a formal suggestion of a plan for how we cope with this: >> >> 1. Implement online upgrade in 9.4 via the various facilities we have >> in-progress. That looks completely possible. >> >> 2. Name the next release after that 10.0 (would have been 9.5). We >> declare now that >> a) 10.0 will support on-line upgrade from 9.4 (only) >> b) various major incompatibilities will be introduced in 10.0 - the >> change in release number will indicate to everybody that is the case >> c) agree that there will be no pg_upgrade patch from 9.4 to 10.0, so >> that we will not be constrained by that > > Assuming online upgrade is going to require logical replication, you are > also assuming 2x storage as you need to have a second cluster to perform > the upgrade. The people that want online upgrade already have 1+ other systems to do this with. > pg_upgrade would still be needed to upgrade a cluster > in-place. > This sounds like, "I created a new tool which does some of what the old > tool does. Let's break the old tool to allow some unspecified changes I > might want to make." I haven't argued against pg_upgrade in general, nor said anything about breaking it. I proposed that we don't support a pg_upgrade path between two near-future releases, as a way of introducing incompatibilities. After that, we would continue to use pg_upgrade for later releases. Logical replication is being developed, which gives us a complete code path for doing what we'd need to do. The most important thing is we wouldn't need to develop any other code that exists just for upgrade. Writing special code just for pg_upgrade will take a lot of work. Running that code would mean pg_upgrade would touch the actual database, which would be down for a long time while it runs. And if it hits a bug during or after, you're hosed. So you'd need to take a full backup before you started the process, probably storing it on disk somewhere and so you would need x2 disk space with this route also. Specialised code is less well tested, which means bugs are more likely to occur and tends to perform more poorly. Not only that, but the first person to want an incompatibility gets to write all the code needed and take responsibility for the bugs. I can't comment for others, but I can say I would not personally choose that route - it looks both expensive and risky. > I consider this thread to be not thought-through, obviously. My proposal has had lots of serious consideration, but that is not the topic of this thread. The title of the thread is a general one, with a clear objective. I'm looking for a way forwards that allows us to introduce the changes that many have proposed and which regrettably result in incompatibilities. If we have no plan I think its likely it will never happen and it is currently blocking useful change. Please explain what you consider to be a better plan, so we can judge all proposals together. --Simon Riggs http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
On Sun, May 26, 2013 at 10:53:37AM +0100, Simon Riggs wrote: > > I consider this thread to be not thought-through, obviously. > > My proposal has had lots of serious consideration, but that is not the > topic of this thread. > > The title of the thread is a general one, with a clear objective. > > I'm looking for a way forwards that allows us to introduce the changes > that many have proposed and which regrettably result in > incompatibilities. If we have no plan I think its likely it will never > happen and it is currently blocking useful change. > > Please explain what you consider to be a better plan, so we can judge > all proposals together. I agree with the idea of using logical replication as a way to do pg_upgrade version-breaking releases. What I don't know is what incompatible changes are pending that would require this. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. +
On Sun, May 26, 2013 at 09:18:11AM -0400, Bruce Momjian wrote: > On Sun, May 26, 2013 at 10:53:37AM +0100, Simon Riggs wrote: > > > I consider this thread to be not thought-through, obviously. > > > > My proposal has had lots of serious consideration, but that is not the > > topic of this thread. > > > > The title of the thread is a general one, with a clear objective. > > > > I'm looking for a way forwards that allows us to introduce the changes > > that many have proposed and which regrettably result in > > incompatibilities. If we have no plan I think its likely it will never > > happen and it is currently blocking useful change. > > > > Please explain what you consider to be a better plan, so we can judge > > all proposals together. > > I agree with the idea of using logical replication as a way to do > pg_upgrade version-breaking releases. What I don't know is what > incompatible changes are pending that would require this. Sorry I was unclear. When I said "not thought-through", I meant, you need to start with the _reason_ we need to break pg_upgrade in an upcoming version, then we can start to plan how to do it. The logical replication idea is a good one for getting us through pg_upgrade version-breaking releases. I am fine with breaking pg_upgrade, but I just don't see the pending reason at this point. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. +
On 05/26/2013 04:22 PM, Bruce Momjian wrote: > On Sun, May 26, 2013 at 09:18:11AM -0400, Bruce Momjian wrote: >> On Sun, May 26, 2013 at 10:53:37AM +0100, Simon Riggs wrote: >>>> I consider this thread to be not thought-through, obviously. >>> My proposal has had lots of serious consideration, but that is not the >>> topic of this thread. >>> >>> The title of the thread is a general one, with a clear objective. >>> >>> I'm looking for a way forwards that allows us to introduce the changes >>> that many have proposed and which regrettably result in >>> incompatibilities. If we have no plan I think its likely it will never >>> happen and it is currently blocking useful change. >>> >>> Please explain what you consider to be a better plan, so we can judge >>> all proposals together. >> I agree with the idea of using logical replication as a way to do >> pg_upgrade version-breaking releases. What I don't know is what >> incompatible changes are pending that would require this. > Sorry I was unclear. When I said "not thought-through", I meant, you > need to start with the _reason_ we need to break pg_upgrade in an > upcoming version, then we can start to plan how to do it. The logical > replication idea is a good one for getting us through pg_upgrade > version-breaking releases. > > I am fine with breaking pg_upgrade, but I just don't see the pending > reason at this point. Not sure which ones Simon meant, but at least any new/better storage manager would seem to me to be requiring a non-pg_upgrade upgrade path unless we require the storage manager to also include a parallel implementation of pg_upgrade. The family of possible storage magers here would include column stores, distributed / partitioned / replicated memory-only / index-structured / ... storages which all could have advantages in certain situations and whic all need an upgrade path. While you could do this using sequance of first pg_upgrading and then doing some internal data migration to new storage manager doing this in one go would be much smoother. -- Hannu Krosing PostgreSQL Consultant Performance, Scalability and High Availability 2ndQuadrant Nordic OÜ
> Not sure which ones Simon meant, but at least any new/better > storage manager would seem to me to be requiring > a non-pg_upgrade upgrade path unless we require the storage manager > to also include a parallel implementation of pg_upgrade. Isn't this a bit of horse-cart inversion here? We just hashed out a tentative, incomplete pseudo-spec for storage managers *yesterday*. We don't have a complete spec at this point, let alone a development plan, and it's entirely possible that we'll be able to implement SMs without breaking pgupgrade. It's also not at all clear that we can develop SMs in less than 2 years.I tend to think it unlikely. First, let's have a few features for which breaking binary compatibility is a necessity or a clear benefit. Then we'll schedule when to break them. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com
* Josh Berkus (josh@agliodbs.com) wrote:
> and it's entirely possible that we'll be able to implement SMs without
> breaking pgupgrade.
I'd certainly hope so.. It's certainly not obvious, to me at least,
why a new SM or supporting any of those features would require
breaking pg_upgrade. Perhaps there's something I'm not seeing there,
but it had better be a *really* good reason..
btw, has anyone posted the SM API proposal..? Unfortunately, I think I
had to leave before that was hashed out..
> First, let's have a few features for which breaking binary compatibility
> is a necessity or a clear benefit. Then we'll schedule when to break them.
Agreed.
Thanks,
Stephen
Stephen Frost <sfrost@snowman.net> writes:
> btw, has anyone posted the SM API proposal..? Unfortunately, I think I
> had to leave before that was hashed out..
There isn't one yet. We think we understand where the pain points are,
but there's still a long way to go to have a proposal.
regards, tom lane
The assumption that we ought to plan expressly for an incompatibility that essentially discards pg_upgrade seems premature, particularly in advance of would-be solutions that, in some cases, mightn't actually work.
If pg_upgrade doesn't work, then, at present, the plausible solutions are to either dump and restore, which might take way too long, or use one of the logical replication systems (e.g. - Slony, Londiste, or similar, in the absence of the would-be built-in logical replication).On 05/25/2013 05:39 PM, Simon Riggs wrote: > 2. Name the next release after that 10.0 (would have been 9.5). We > declare now that > a) 10.0 will support on-line upgrade from 9.4 (only) > b) various major incompatibilities will be introduced in 10.0 - the > change in release number will indicate to everybody that is the case > c) agree that there will be no pg_upgrade patch from 9.4 to 10.0, so > that we will not be constrained by that While we're talking about changing things, what about: - Switching to single-major-version release numbering. The number of people who say "PostgreSQL 9.x" is amazing; even *packagers* get this wrong and produce "postgresql-9" packages. Witness Amazon Linux's awful PostgreSQL packages for example. Going to PostgreSQL 10.0, 11.0, 12.0, etc with a typical major/minor scheme might be worth considering. - s/cluster/server/g . Just because "cluster" is historical usage doesn't make it any less confusing for users. *dives for asbestos fire suit* -- Craig Ringer http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
On Mon, May 27, 2013 at 2:01 PM, Craig Ringer <craig@2ndquadrant.com> wrote:
--
Michael
On 05/25/2013 05:39 PM, Simon Riggs wrote:- Switching to single-major-version release numbering. The number of
people who say "PostgreSQL 9.x" is amazing; even *packagers* get this
wrong and produce "postgresql-9" packages. Witness Amazon Linux's awful
PostgreSQL packages for example. Going to PostgreSQL 10.0, 11.0, 12.0,
etc with a typical major/minor scheme might be worth considering.
In this case you don't even need the 2nd digit...
Btw, -1 for the idea, as it would remove the possibility to tell that a new major release incrementing the 1st digit of version number brings more enhancement than normal major releases incrementing the 1st digit. This was the case for 9.0, helping people in remembering that streaming replication has been introduced from 9.x series.Michael
On Sun, May 26, 2013 at 09:18:41PM -0400, Stephen Frost wrote: > * Josh Berkus (josh@agliodbs.com) wrote: > > and it's entirely possible that we'll be able to implement SMs without > > breaking pgupgrade. > > I'd certainly hope so.. It's certainly not obvious, to me at least, > why a new SM or supporting any of those features would require > breaking pg_upgrade. Perhaps there's something I'm not seeing there, > but it had better be a *really* good reason.. If I had to _guess_, I would say users who are using the default storage manager would still be able to use pg_upgrade, and those using non-default storage managers perhaps can't. But, again, this is all so hypothetical that it doesn't seem worth talking about. My big point is that someone came to me at PGCon asking if I knew anything about why Simon thought we needed to break pg_upgrade in <2 years, and I said no, so I had go digging into my email to find out what was going on. Simon has a very visible position in the community, so when he suggests something, people take it seriously, which means I have to address it. I would prefer if there was more thought put into the ideas before they are posted. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. +
* Bruce Momjian (bruce@momjian.us) wrote:
> If I had to _guess_, I would say users who are using the default storage
> manager would still be able to use pg_upgrade, and those using
> non-default storage managers perhaps can't.
That would make sense.
> But, again, this is all so hypothetical that it doesn't seem worth
> talking about.
Having a specific list of "these are the things we want to change, and
why, and here is why pg_upgrade can't support it" would be much more
useful to work from, I agree.
That said, many discussions and ideas do get shut down, perhaps too
early, because of pg_upgrade considerations. If we had a plan to have
an incompatible release in the future, those ideas and discussions might
be able to progress to a point where we determine it's worth it to take
the pain of a non-pg_upgrade-supported release. That's a bit of a
stretch, in my view, but I suppose it's possible. Even so though, I
would suggest that we put together a wiki page to list out those items
and encourage people to add to such a list; perhaps having an item on
that list would make discussion about it progress beyond "it breaks
pg_upgrade".
Thanks,
Stephen
On Mon, May 27, 2013 at 08:26:48AM -0400, Stephen Frost wrote: > * Bruce Momjian (bruce@momjian.us) wrote: > > If I had to _guess_, I would say users who are using the default storage > > manager would still be able to use pg_upgrade, and those using > > non-default storage managers perhaps can't. > > That would make sense. > > > But, again, this is all so hypothetical that it doesn't seem worth > > talking about. > > Having a specific list of "these are the things we want to change, and > why, and here is why pg_upgrade can't support it" would be much more > useful to work from, I agree. > > That said, many discussions and ideas do get shut down, perhaps too > early, because of pg_upgrade considerations. If we had a plan to have > an incompatible release in the future, those ideas and discussions might > be able to progress to a point where we determine it's worth it to take > the pain of a non-pg_upgrade-supported release. That's a bit of a > stretch, in my view, but I suppose it's possible. Even so though, I > would suggest that we put together a wiki page to list out those items > and encourage people to add to such a list; perhaps having an item on > that list would make discussion about it progress beyond "it breaks > pg_upgrade". Yes, we should be collecting things we want to do for a pg_upgrade break so we can see the list all in one place. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. +
On 05/26/2013 06:18 PM, Josh Berkus wrote: >> Not sure which ones Simon meant, but at least any new/better >> storage manager would seem to me to be requiring >> a non-pg_upgrade upgrade path unless we require the storage manager >> to also include a parallel implementation of pg_upgrade. > Isn't this a bit of horse-cart inversion here? We just hashed out a > tentative, incomplete pseudo-spec for storage managers *yesterday*. Many people have been *thinking* about pluggable storage / storage managers for much longer time. > We > don't have a complete spec at this point, let alone a development plan, I think we will have a development plan *before* complete spec anyway :) > and it's entirely possible that we'll be able to implement SMs without > breaking pgupgrade. My point was exactly to not spend majority of new storage manager discussion on "does it break pg_upgrade", "maybe we can find a way to do it without breaking pg_upgrade", etc... > It's also not at all clear that we can develop SMs in less than 2 years. > I tend to think it unlikely. I think the important part of Simons message was not "two years" > First, let's have a few features for which breaking binary compatibility > is a necessity or a clear benefit. Then we'll schedule when to break them. But rather than "it breaks pg_upgrade" not being a complete stopper for proposed useful features that might break it. -- Hannu Krosing PostgreSQL Consultant Performance, Scalability and High Availability 2ndQuadrant Nordic OÜ
Bruce Momjian <bruce@momjian.us> writes:
> On Mon, May 27, 2013 at 08:26:48AM -0400, Stephen Frost wrote:
>> That said, many discussions and ideas do get shut down, perhaps too
>> early, because of pg_upgrade considerations. If we had a plan to have
>> an incompatible release in the future, those ideas and discussions might
>> be able to progress to a point where we determine it's worth it to take
>> the pain of a non-pg_upgrade-supported release. That's a bit of a
>> stretch, in my view, but I suppose it's possible. Even so though, I
>> would suggest that we put together a wiki page to list out those items
>> and encourage people to add to such a list; perhaps having an item on
>> that list would make discussion about it progress beyond "it breaks
>> pg_upgrade".
> Yes, we should be collecting things we want to do for a pg_upgrade break
> so we can see the list all in one place.
Precisely. We've said right along that we reserve the right to have a
non-upgradable disk format change whenever sufficiently many reasons
accumulate to do that. The way to go about that is to collect projects
that need to be kept on hold for such a release --- not to say we're
going to have such a release and then look for reasons.
regards, tom lane
Michael Paquier escribió: > On Mon, May 27, 2013 at 2:01 PM, Craig Ringer <craig@2ndquadrant.com> wrote: > > > On 05/25/2013 05:39 PM, Simon Riggs wrote: > > - Switching to single-major-version release numbering. The number of > > people who say "PostgreSQL 9.x" is amazing; even *packagers* get this > > wrong and produce "postgresql-9" packages. Witness Amazon Linux's awful > > PostgreSQL packages for example. Going to PostgreSQL 10.0, 11.0, 12.0, > > etc with a typical major/minor scheme might be worth considering. > > > In this case you don't even need the 2nd digit... You do -- they are used for minor releases, i.e. 10.1 would be a bugfix release for 10.0. If we continue using the current numbering scheme, 10.1 would be the major version after 10.0. > Btw, -1 for the idea, as it would remove the possibility to tell that a new > major release incrementing the 1st digit of version number brings more > enhancement than normal major releases incrementing the 1st digit. This was > the case for 9.0, helping people in remembering that streaming replication > has been introduced from 9.x series. All major releases bring lots of enhancements. Streaming replication might be great for some people, but I'm sure there are things in 8.4 and 9.1 that are equally great for some other people. -- Álvaro Herrera http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
Tom Lane wrote: > Bruce Momjian <bruce@momjian.us> writes: > > Yes, we should be collecting things we want to do for a pg_upgrade break > > so we can see the list all in one place. > > Precisely. We've said right along that we reserve the right to have a > non-upgradable disk format change whenever sufficiently many reasons > accumulate to do that. Do we have a wiki page about this? -- Álvaro Herrera http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
On Mon, May 27, 2013 at 09:17:50AM -0400, Bruce Momjian wrote: > > That said, many discussions and ideas do get shut down, perhaps too > > early, because of pg_upgrade considerations. If we had a plan to have > > an incompatible release in the future, those ideas and discussions might > > be able to progress to a point where we determine it's worth it to take > > the pain of a non-pg_upgrade-supported release. That's a bit of a > > stretch, in my view, but I suppose it's possible. Even so though, I > > would suggest that we put together a wiki page to list out those items > > and encourage people to add to such a list; perhaps having an item on > > that list would make discussion about it progress beyond "it breaks > > pg_upgrade". > > Yes, we should be collecting things we want to do for a pg_upgrade break > so we can see the list all in one place. OK, I have added a section to the TODO list for this: Desired changes that would prevent upgrades with pg_upgrade 32-bit page checksums Are there any others? -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. +
On 27 May 2013 15:36, Tom Lane <tgl@sss.pgh.pa.us> wrote: > Bruce Momjian <bruce@momjian.us> writes: >> On Mon, May 27, 2013 at 08:26:48AM -0400, Stephen Frost wrote: >>> That said, many discussions and ideas do get shut down, perhaps too >>> early, because of pg_upgrade considerations. If we had a plan to have >>> an incompatible release in the future, those ideas and discussions might >>> be able to progress to a point where we determine it's worth it to take >>> the pain of a non-pg_upgrade-supported release. That's a bit of a >>> stretch, in my view, but I suppose it's possible. Even so though, I >>> would suggest that we put together a wiki page to list out those items >>> and encourage people to add to such a list; perhaps having an item on >>> that list would make discussion about it progress beyond "it breaks >>> pg_upgrade". > >> Yes, we should be collecting things we want to do for a pg_upgrade break >> so we can see the list all in one place. > > Precisely. We've said right along that we reserve the right to have a > non-upgradable disk format change whenever sufficiently many reasons > accumulate to do that. I'm happy with that. I was also thinking about collecting changes not related just to disk format, if any exist. > The way to go about that is to collect projects > that need to be kept on hold for such a release --- not to say we're > going to have such a release and then look for reasons. Agreed. I was trying to establish a realistic timeline for such events, so that the planning was able to be taken seriously. Yes, it wass a "work backwards" or "what if" type of planning. But now we have a rough plan of how it might look, collecting ideas can begin. --Simon Riggs http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
Bruce Momjian <bruce@momjian.us> writes:
> On Mon, May 27, 2013 at 09:17:50AM -0400, Bruce Momjian wrote:
>> Yes, we should be collecting things we want to do for a pg_upgrade break
>> so we can see the list all in one place.
> OK, I have added a section to the TODO list for this:
> Desired changes that would prevent upgrades with pg_upgrade
> 32-bit page checksums
> Are there any others?
GiST indexes really oughta have a metapage so there can be a version
number in them.
Also, if we are going to unify hstore and json, it'd be nice if we could
change the existing binary representation of hstore (per discussions at
Oleg and Teodor's talk --- this will be moot if we invent a new core
type, but it'd be better not to have to).
There are probably some other data-type-specific cleanups we could
make, but I have to go get on an airplane so no time to think about it.
regards, tom lane
Bruce Momjian wrote: > OK, I have added a section to the TODO list for this: > > Desired changes that would prevent upgrades with pg_upgrade > > 32-bit page checksums > > Are there any others? I would have each data segment be self-identifying, i.e. have a magic number at the beginning of the file and the relation OID, some fork identification and the segment number somewhere -- probably the special space of the first page. -- Álvaro Herrera http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
On Tue, May 28, 2013 at 12:36 AM, Alvaro Herrera <alvherre@2ndquadrant.com> wrote:
-- Michael Paquier escribió:> On Mon, May 27, 2013 at 2:01 PM, Craig Ringer <craig@2ndquadrant.com> wrote:You do -- they are used for minor releases, i.e. 10.1 would be a bugfix
>
> > On 05/25/2013 05:39 PM, Simon Riggs wrote:
> > - Switching to single-major-version release numbering. The number of
> > people who say "PostgreSQL 9.x" is amazing; even *packagers* get this
> > wrong and produce "postgresql-9" packages. Witness Amazon Linux's awful
> > PostgreSQL packages for example. Going to PostgreSQL 10.0, 11.0, 12.0,
> > etc with a typical major/minor scheme might be worth considering.
> >
> In this case you don't even need the 2nd digit...
release for 10.0. If we continue using the current numbering scheme,
10.1 would be the major version after 10.0.
Sorry for the confusion. I meant that the 2nd digit would not be necessary when identifying a given major release, so I just didn't get the meaning of what Craig said. As you say, you would still need the 2nd digit for minor releases.
Michael
On Tue, May 28, 2013 at 07:39:35AM +0900, Michael Paquier wrote: > On Tue, May 28, 2013 at 12:36 AM, Alvaro Herrera > <alvherre@2ndquadrant.com>wrote: > > > Michael Paquier escribió: > > > On Mon, May 27, 2013 at 2:01 PM, Craig Ringer <craig@2ndquadrant.com> > > wrote: > > > > > > > On 05/25/2013 05:39 PM, Simon Riggs wrote: > > > > - Switching to single-major-version release numbering. The number of > > > > people who say "PostgreSQL 9.x" is amazing; even *packagers* get this > > > > wrong and produce "postgresql-9" packages. Witness Amazon Linux's awful > > > > PostgreSQL packages for example. Going to PostgreSQL 10.0, 11.0, 12.0, > > > > etc with a typical major/minor scheme might be worth considering. > > > > > > > In this case you don't even need the 2nd digit... > > > > You do -- they are used for minor releases, i.e. 10.1 would be a bugfix > > release for 10.0. If we continue using the current numbering scheme, > > 10.1 would be the major version after 10.0. > > > Sorry for the confusion. I meant that the 2nd digit would not be necessary > when identifying a given major release, so I just didn't get the meaning of > what Craig said. As you say, you would still need the 2nd digit for minor > releases. What's been proposed before that wouldn't break previous applications is a numbering system like this: 10.0.0 10.0.1 10.0.2 10.0.3 ... 11.0.0 11.0.1 i.e. only change the "most-major" version number and always leave the "less-major" number as zero. Cheers, David. -- David Fetter <david@fetter.org> http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fetter@gmail.com iCal: webcal://www.tripit.com/feed/ical/people/david74/tripit.ics Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate
Michael Paquier escribió: > On Tue, May 28, 2013 at 12:36 AM, Alvaro Herrera > <alvherre@2ndquadrant.com>wrote: > > You do -- they are used for minor releases, i.e. 10.1 would be a bugfix > > release for 10.0. If we continue using the current numbering scheme, > > 10.1 would be the major version after 10.0. > > > Sorry for the confusion. I meant that the 2nd digit would not be necessary > when identifying a given major release, so I just didn't get the meaning of > what Craig said. As you say, you would still need the 2nd digit for minor > releases. Well, that seems okay to me. We used to see a lot of people talking about "Postgres 8.x" when they meant, say, 8.3; and now we have people talking about "Postgres 9" when in reality they mean 9.1 or some other specific major version. Having the second digit be part of the major version number is a difficult idea to convey to external people. -- Álvaro Herrera http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
On Tue, May 28, 2013 at 7:52 AM, David Fetter <david@fetter.org> wrote:
--
Michael
is a numbering system like this:What's been proposed before that wouldn't break previous applications
10.0.0
10.0.1
10.0.2
10.0.3
...
11.0.0
11.0.1
i.e. only change the "most-major" version number and always leave the
"less-major" number as zero.
Thanks for the clarification. Firefox did exactly the same from 4.0.
Michael
<div class="moz-cite-prefix">On 05/27/2013 05:45 PM, Michael Paquier wrote:<br /></div><blockquote cite="mid:CAB7nPqQtwgrVMEgPuKNGyVZCYZTWSg7NY9G46XcGZy0-Nh3-Rg@mail.gmail.com"type="cite"><pre wrap="">On Mon, May 27, 2013at 2:01 PM, Craig Ringer <a class="moz-txt-link-rfc2396E" href="mailto:craig@2ndquadrant.com"><craig@2ndquadrant.com></a>wrote: </pre><blockquote type="cite"><pre wrap="">On 05/25/2013 05:39 PM, Simon Riggs wrote: - Switching to single-major-version release numbering. The number of people who say "PostgreSQL 9.x" is amazing; even *packagers* get this wrong and produce "postgresql-9" packages. Witness Amazon Linux's awful PostgreSQL packages for example. Going to PostgreSQL 10.0, 11.0, 12.0, etc with a typical major/minor scheme might be worth considering. </pre></blockquote><pre wrap="">In this case you don't even need the 2nd digit... Btw, -1 for the idea, as it would remove the possibility to tell that a new major release incrementing the 1st digit of version number brings more enhancement than normal major releases incrementing the 1st digit. This was the case for 9.0, helping people in remembering that streaming replication has been introduced from 9.x series.</pre></blockquote> I don't find bumping the major to be particularly helpful. Everyrelease brings major features - and some introduce major incompatibilities.<br /><br /> 8.4 introduced CTEs.<br /> 8.3broke tons of client code with the removal of implicit casts to text.<br /><br /> It really depends on what features youconsider more major/significant. Personally I don't think it makes sense to try to say "this release is bigger" in Pg- at least not in terms of enhancement. I can see value in using this-release-is-bigger for "this brings more breakage"- but would strongly prefer a smooth and continuous release numbering that doesn't confuse the heck out of users.<br/><br /> I'm extremely tired of being told "I'm running PostgreSQL 8.x" or "I'm running PostgreSQL 9.x" and havingto point out the version policy, the fact that there are four years and huge fixes/enhancements between 8.0 and 8.4,etc.<br /><br /> The version policy makes _no distinction_ between which digit changes in a major release:<br /><br />"PostgreSQL major releases include new features and occur roughly once every year. A major release is numbered by increasingeither the first or second part of the version number, e.g. 8.2 to 8.3.<br /><br /> "Major releases usually changethe internal format of system tables and data files. These changes are often complex, so we do not maintain backwardcompatibility of all stored data. A dump/reload of the database or use of the pg_upgrade module is required for majorupgrades."<br /><br /> and I strongly believe that we should drop the notion entirely.<br /><br /> ... <br /><br /><preclass="moz-signature" cols="72">-- Craig Ringer <a class="moz-txt-link-freetext" href="http://www.2ndQuadrant.com/">http://www.2ndQuadrant.com/</a>PostgreSQLDevelopment, 24x7 Support, Training & Services</pre>
On 05/28/2013 07:22 AM, Michael Paquier wrote: > On Tue, May 28, 2013 at 7:52 AM, David Fetter <david@fetter.org> wrote: > >> What's been proposed before that wouldn't break previous applications >> is a numbering system like this: >> >> 10.0.0 >> 10.0.1 >> 10.0.2 >> 10.0.3 >> ... >> 11.0.0 >> 11.0.1 >> >> i.e. only change the "most-major" version number and always leave the >> "less-major" number as zero. >> > Thanks for the clarification. Firefox did exactly the same from 4.0. Yeah... I was more meaning 10.0, 10.1, 10.2 etc for minor releases, but I can imagine people coding logic to check "major version" using the first two digits, so you're quite right that it'd need to be grandfathered into 10.0.1, 10.0.2, etc. Sigh. The upside of that is that it'd reinforce the idea that we sometimes struggle to get across to people - that minor patch releases are *minor* and *safe* to just upgrade to without jumping through change-approval hoops, vendor approval for updates, two-year-long QA and all the other baggage many IT departments seem to have. -- Craig Ringer http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
On 05/28/2013 12:41 AM, Simon Riggs wrote: > I'm happy with that. > > I was also thinking about collecting changes not related just to disk > format, if any exist. Any wire protocol or syntax changes? I can't seem to find a "things we want to do in wire protocol v4" doc in the wiki but I know I've seen occasional discussion of things that can't be done without protocol changes. Anyone with a better memory than me able to pitch in? What'd be required to support in-band query cancellation? Sending per-statement GUCs (to allow true statement timeout)? I can't think of any major syntax warts and grandfathered quirks that'd be really great to get rid of if we had the freedom to break things. -- Craig Ringer http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
On 05/27/2013 04:58 PM, Craig Ringer wrote: > > On 05/28/2013 12:41 AM, Simon Riggs wrote: >> I'm happy with that. >> >> I was also thinking about collecting changes not related just to disk >> format, if any exist. > Any wire protocol or syntax changes? > > I can't seem to find a "things we want to do in wire protocol v4" doc in > the wiki but I know I've seen occasional discussion of things that can't > be done without protocol changes. Anyone with a better memory than me > able to pitch in? > > What'd be required to support in-band query cancellation? Sending > per-statement GUCs (to allow true statement timeout)? > I would like to see the ability to define if a query is read only at the protocol level, so that load balances that speak libpq can know what to do with the query without parsing it. JD -- Command Prompt, Inc. - http://www.commandprompt.com/ 509-416-6579 PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC, @cmdpromptinc For my dreams of your image that blossoms a rose in the deeps of my heart. - W.B. Yeats
<div class="moz-cite-prefix">On 28/05/13 11:48, Craig Ringer wrote:<br /></div><blockquote cite="mid:51A3F0C9.90600@2ndquadrant.com"type="cite"><div class="moz-cite-prefix">On 05/27/2013 05:45 PM, Michael Paquierwrote:<br /></div><blockquote cite="mid:CAB7nPqQtwgrVMEgPuKNGyVZCYZTWSg7NY9G46XcGZy0-Nh3-Rg@mail.gmail.com" type="cite"><prewrap="">On Mon, May 27, 2013 at 2:01 PM, Craig Ringer <a class="moz-txt-link-rfc2396E" href="mailto:craig@2ndquadrant.com"moz-do-not-send="true"><craig@2ndquadrant.com></a> wrote: </pre><blockquote type="cite"><pre wrap="">On 05/25/2013 05:39 PM, Simon Riggs wrote: - Switching to single-major-version release numbering. The number of people who say "PostgreSQL 9.x" is amazing; even *packagers* get this wrong and produce "postgresql-9" packages. Witness Amazon Linux's awful PostgreSQL packages for example. Going to PostgreSQL 10.0, 11.0, 12.0, etc with a typical major/minor scheme might be worth considering. </pre></blockquote><pre wrap="">In this case you don't even need the 2nd digit... Btw, -1 for the idea, as it would remove the possibility to tell that a new major release incrementing the 1st digit of version number brings more enhancement than normal major releases incrementing the 1st digit. This was the case for 9.0, helping people in remembering that streaming replication has been introduced from 9.x series.</pre></blockquote> I don't find bumping the major to be particularly helpful. Everyrelease brings major features - and some introduce major incompatibilities.<br /><br /> 8.4 introduced CTEs.<br /> 8.3broke tons of client code with the removal of implicit casts to text.<br /><br /> It really depends on what features youconsider more major/significant. Personally I don't think it makes sense to try to say "this release is bigger" in Pg- at least not in terms of enhancement. I can see value in using this-release-is-bigger for "this brings more breakage"- but would strongly prefer a smooth and continuous release numbering that doesn't confuse the heck out of users.<br/><br /> I'm extremely tired of being told "I'm running PostgreSQL 8.x" or "I'm running PostgreSQL 9.x" and havingto point out the version policy, the fact that there are four years and huge fixes/enhancements between 8.0 and 8.4,etc.<br /><br /> The version policy makes _no distinction_ between which digit changes in a major release:<br /><br />"PostgreSQL major releases include new features and occur roughly once every year. A major release is numbered by increasingeither the first or second part of the version number, e.g. 8.2 to 8.3.<br /><br /> "Major releases usually changethe internal format of system tables and data files. These changes are often complex, so we do not maintain backwardcompatibility of all stored data. A dump/reload of the database or use of the pg_upgrade module is required for majorupgrades."<br /><br /> and I strongly believe that we should drop the notion entirely.<br /><br /> ... <br /><br /><preclass="moz-signature" cols="72">-- Craig Ringer <a class="moz-txt-link-freetext" href="http://www.2ndQuadrant.com/"moz-do-not-send="true">http://www.2ndQuadrant.com/</a>PostgreSQL Development, 24x7 Support,Training & Services</pre></blockquote><font size="-1">Yes, I hate the Firefox style number inflati<font size="-1">on.<br/><br /><br /><font size="-1">Cheers,<br /><font size="-1">Gavin<br /><br /></font></font></font></font>
On 05/28/2013 09:39 AM, Gavin Flower wrote: > Yes, I hate the Firefox style number inflation. I was arguing *for* it ;-) I don't like it much either, but (a) we do about one release a year, not one every few weeks and (b) it's very clear from a quick look at Stack Overflow or first-posts to pgsql-general how confusing two-part major versions are to users. If it's a bit less aesthetically pleasing I'm OK with that. -- Craig Ringer http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
On 05/27/2013 06:53 PM, Craig Ringer wrote: > > On 05/28/2013 09:39 AM, Gavin Flower wrote: >> Yes, I hate the Firefox style number inflation. > I was arguing *for* it ;-) > > I don't like it much either, but (a) we do about one release a year, not > one every few weeks and (b) it's very clear from a quick look at Stack > Overflow or first-posts to pgsql-general how confusing two-part major > versions are to users. If it's a bit less aesthetically pleasing I'm OK > with that. > This argument comes up every couple of years and the people that are trying to solve the problem by changing the versioning are ignoring the fact that there is no problem to solve. Consider the following exchange: Client: I have X problem with PostgreSQL CMD: What version? Client: 9 CMD: Which version of 9? Client: 9.0.2 CMD: You should be running 9.2.4 or at least 9.0.13 Now, if we change the version numbers: Client: I have X problem with PostgreSQL CMD: What version? Client: 9 CMD: Which version of 9? Client: 9.0.2 CMD: You should be running 10.0.5 or at least 9.0.13 The conversation does not change. Further, we are not Firefox. We are not user software. We are developer software. Sincerely, Joshua D. Drake
On Sat, May 25, 2013 at 11:27 AM, Merlin Moncure <mmoncure@gmail.com> wrote: > On Sat, May 25, 2013 at 4:39 AM, Simon Riggs <simon@2ndquadrant.com> wrote: >> There are a number of changes we'd probably like to make to the way >> things work in Postgres. This thread is not about discussing what >> those are, just to say that requirements exist and have been discussed >> in various threads over time. >> >> The constraint on such changes is that we've decided that we must have >> an upgrade path from release to release. >> >> So I'd like to make a formal suggestion of a plan for how we cope with this: >> >> 1. Implement online upgrade in 9.4 via the various facilities we have >> in-progress. That looks completely possible. >> >> 2. Name the next release after that 10.0 (would have been 9.5). We >> declare now that >> a) 10.0 will support on-line upgrade from 9.4 (only) >> b) various major incompatibilities will be introduced in 10.0 - the >> change in release number will indicate to everybody that is the case >> c) agree that there will be no pg_upgrade patch from 9.4 to 10.0, so >> that we will not be constrained by that >> >> This plan doesn't presume any particular change. Each change would >> need to be discussed on a separate thread, with a separate case for >> each. All I'm suggesting is that we have a coherent plan for the >> timing of such changes, so we can bundle them together into one >> release. >> >> By doing this now we give ourselves lots of time to plan changes that >> will see us good for another decade. If we don't do this, then we >> simply risk losing the iniative by continuing to support legacy >> formats and approaches. > > Huh. I don't think that bumping the version number to 10.0 vs 9.5 is > justification to introduce breaking changes. In fact, I would rather > see 10.0 be the version where we formally stop doing that. I > understand that some stuff needs to be improved but it often doesn't > seem to be worth the cost in the long run. Please disregard this comment -- I didn't realize the topic was regarding on disk format -- I mistakenly though it was opening the door for user level feature changes. merlin
On 05/28/2013 06:13 AM, Joshua D. Drake wrote: > > On 05/27/2013 06:53 PM, Craig Ringer wrote: >> >> On 05/28/2013 09:39 AM, Gavin Flower wrote: >>> Yes, I hate the Firefox style number inflation. >> I was arguing *for* it ;-) >> >> I don't like it much either, but (a) we do about one release a year, not >> one every few weeks and (b) it's very clear from a quick look at Stack >> Overflow or first-posts to pgsql-general how confusing two-part major >> versions are to users. If it's a bit less aesthetically pleasing I'm OK >> with that. >> > > This argument comes up every couple of years and the people that > are trying to solve the problem by changing the versioning are > ignoring the fact that there is no problem to solve. > > Consider the following exchange: > > Client: I have X problem with PostgreSQL > CMD: What version? > Client: 9 > CMD: Which version of 9? > Client: 9.0.2 > CMD: You should be running 9.2.4 or at least 9.0.13 If the problem has the "at least" part, then the first part is superfluous. If somebody wants to figure out how to run streaming CTE-s on "postgresql 8" then you need to ask for exact "major version which is two first digits" if they want to run streaming replication there you can skip on e-mail exchange and tell right away that SR was added in version 9.0 > ... > > The conversation does not change. > > Further, we are not Firefox. We are not user software. We are > developer software. At least some of the real-world problems with PostgreSQL comes from "We are developer software" mentality. Yes, We are developer software, but we are also a DBA/maintainer/infrastructure manager software which needs to live a long time after the "development" is finished. -- Hannu Krosing PostgreSQL Consultant Performance, Scalability and High Availability 2ndQuadrant Nordic OÜ
>> This argument comes up every couple of years and the people that >> are trying to solve the problem by changing the versioning are >> ignoring the fact that there is no problem to solve. We just had this discussion on -advocacy (where it belongs, frankly) a couple months ago: http://www.postgresql.org/message-id/512E8EF8.3000507@agliodbs.com To sum up: the negatives of changing our version numbering scheme outweighed the positives. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com
On Tue, May 28, 2013 at 11:56 AM, Josh Berkus <josh@agliodbs.com> wrote: > >>> This argument comes up every couple of years and the people that >>> are trying to solve the problem by changing the versioning are >>> ignoring the fact that there is no problem to solve. > > We just had this discussion on -advocacy (where it belongs, frankly) +1. > a > couple months ago: > > http://www.postgresql.org/message-id/512E8EF8.3000507@agliodbs.com > > To sum up: the negatives of changing our version numbering scheme > outweighed the positives. And +1 to that, too. FWIW, I think we may want to consider retitling 9.4 as 10.0, not because of any binary compatibility break (which, for the record, I oppose) but because of features. It's a little early to make that call just yet, of course, but I have a good feeling about this cycle. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
On 05/28/2013 08:36 AM, Hannu Krosing wrote: >> The conversation does not change. >> >> Further, we are not Firefox. We are not user software. We are >> developer software. > At least some of the real-world problems with PostgreSQL > comes from "We are developer software" mentality. > > Yes, We are developer software, but we are also a > DBA/maintainer/infrastructure manager I would not hire any of those three that weren't smart enough to understand our versioning scheme or had the wits to open a web browser and google: PostgreSQL versioning The answer is link #1 on Google. That said, I won't raise a stink. I am not really of a strong opinion either way except to say we are not solving a problem. We are just tickling each other's fancies. JD -- Command Prompt, Inc. - http://www.commandprompt.com/ 509-416-6579 PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC, @cmdpromptinc For my dreams of your image that blossoms a rose in the deeps of my heart. - W.B. Yeats
On Tue, May 28, 2013 at 07:58:33AM +0800, Craig Ringer wrote: > On 05/28/2013 12:41 AM, Simon Riggs wrote: > > I'm happy with that. > > > > I was also thinking about collecting changes not related just to disk > > format, if any exist. > Any wire protocol or syntax changes? > > I can't seem to find a "things we want to do in wire protocol v4" doc in > the wiki but I know I've seen occasional discussion of things that can't > be done without protocol changes. Anyone with a better memory than me > able to pitch in? Sure, it is on the TODO list: https://wiki.postgresql.org/wiki/Todo#.2Fcontrib.2Fpg_upgrade I can only get a link to pg_upgrade from there, so look two sections below that for "Wire Protocol Changes". -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. +
On Mon, May 27, 2013 at 05:21:16PM -0700, Joshua D. Drake wrote: > > On 05/27/2013 04:58 PM, Craig Ringer wrote: > > > >On 05/28/2013 12:41 AM, Simon Riggs wrote: > >>I'm happy with that. > >> > >>I was also thinking about collecting changes not related just to disk > >>format, if any exist. > >Any wire protocol or syntax changes? > > > >I can't seem to find a "things we want to do in wire protocol v4" doc in > >the wiki but I know I've seen occasional discussion of things that can't > >be done without protocol changes. Anyone with a better memory than me > >able to pitch in? > > > >What'd be required to support in-band query cancellation? Sending > >per-statement GUCs (to allow true statement timeout)? > > > > I would like to see the ability to define if a query is read only at > the protocol level, so that load balances that speak libpq can know > what to do with the query without parsing it. Sounds nice, but how would we do that? That would require libpq to know it, right? Do we pass anything back after parsing but before execution?Could it be optional? What about functions thatmodify the database --- isn't that only known at execution time? -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. +
On Mon, May 27, 2013 at 02:09:05PM -0400, Tom Lane wrote: > Bruce Momjian <bruce@momjian.us> writes: > > On Mon, May 27, 2013 at 09:17:50AM -0400, Bruce Momjian wrote: > >> Yes, we should be collecting things we want to do for a pg_upgrade break > >> so we can see the list all in one place. > > > OK, I have added a section to the TODO list for this: > > > Desired changes that would prevent upgrades with pg_upgrade > > 32-bit page checksums > > > Are there any others? > > GiST indexes really oughta have a metapage so there can be a version > number in them. > > Also, if we are going to unify hstore and json, it'd be nice if we could > change the existing binary representation of hstore (per discussions at > Oleg and Teodor's talk --- this will be moot if we invent a new core > type, but it'd be better not to have to). > > There are probably some other data-type-specific cleanups we could > make, but I have to go get on an airplane so no time to think about it. OK, GiST and hstore added to TODO list. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. +
On 05/28/2013 02:18 PM, Bruce Momjian wrote: >> I would like to see the ability to define if a query is read only at >> the protocol level, so that load balances that speak libpq can know >> what to do with the query without parsing it. > > Sounds nice, but how would we do that? That would require libpq to know > it, right? Do we pass anything back after parsing but before execution? > Could it be optional? What about functions that modify the database > --- isn't that only known at execution time? I can't speak to the actual C code that would be required but from a user space, I could see something like this: con = psycopg2.connect(database='testdb', user='test', transaction-type='r') Thus when the connection is made, before anything else is done, we know it is a read only connection and therefore any load balancer speaking libpq would also know it is a read only. The default of course would be r/w and you would use a different connection handler for r/w or w queries. The other option would be to do it on query execute but that doesn't seem as efficient as it would have to be parsed each time. Although it would still be better than reading the actual SQL. Sincerely, Joshua D. Drake -- Command Prompt, Inc. - http://www.commandprompt.com/ 509-416-6579 PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC, @cmdpromptinc For my dreams of your image that blossoms a rose in the deeps of my heart. - W.B. Yeats
Bruce Momjian wrote: > On Mon, May 27, 2013 at 05:21:16PM -0700, Joshua D. Drake wrote: > > I would like to see the ability to define if a query is read only at > > the protocol level, so that load balances that speak libpq can know > > what to do with the query without parsing it. > > Sounds nice, but how would we do that? That would require libpq to know > it, right? Do we pass anything back after parsing but before execution? > Could it be optional? What about functions that modify the database > --- isn't that only known at execution time? Well, if you hit anything that tries to acquire an Xid, and you're in a context that said only read-only was acceptable, just raise an error. In a similar vein, I vaguely recall we discussed (after some security vulnerability involving SQL injection) a mode where we only accept only one command per PQexec() call, i.e. reject execution of commands that contain multiple queries. -- Álvaro Herrera http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
On Tue, May 28, 2013 at 02:26:06PM -0700, Joshua D. Drake wrote: > >Sounds nice, but how would we do that? That would require libpq to know > >it, right? Do we pass anything back after parsing but before execution? > > Could it be optional? What about functions that modify the database > >--- isn't that only known at execution time? > > I can't speak to the actual C code that would be required but from a > user space, I could see something like this: > > con = psycopg2.connect(database='testdb', user='test', > transaction-type='r') > > Thus when the connection is made, before anything else is done, we > know it is a read only connection and therefore any load balancer > speaking libpq would also know it is a read only. The default of > course would be r/w and you would use a different connection handler > for r/w or w queries. > > The other option would be to do it on query execute but that doesn't > seem as efficient as it would have to be parsed each time. Although > it would still be better than reading the actual SQL. Well, you could do SET TRANSACTION READ ONLY, and that would prevent any write transactions. You could assume it is a read query, and get the error and resubmit on the master if that happens, but that sounds inefficient. I thought you were asking for something where you could submit a query and it would report back as read/write or read-only. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. +
On Mon, May 27, 2013 at 03:06:13PM -0400, Alvaro Herrera wrote: > Bruce Momjian wrote: > > > OK, I have added a section to the TODO list for this: > > > > Desired changes that would prevent upgrades with pg_upgrade > > > > 32-bit page checksums > > > > Are there any others? > > I would have each data segment be self-identifying, i.e. have a magic > number at the beginning of the file and the relation OID, some fork > identification and the segment number somewhere -- probably the special > space of the first page. Is this something we want on the TODO? I was not clear how to do with without making the first page format special or wasting space on all the other pages. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. +
On 05/28/2013 03:36 PM, Bruce Momjian wrote: >> The other option would be to do it on query execute but that doesn't >> seem as efficient as it would have to be parsed each time. Although >> it would still be better than reading the actual SQL. > > Well, you could do SET TRANSACTION READ ONLY, and that would prevent any > write transactions. You could assume it is a read query, and get the > error and resubmit on the master if that happens, but that sounds > inefficient. I thought you were asking for something where you could > submit a query and it would report back as read/write or read-only. No I am suggesting something that before anything happens with the parser, the protocol knows what is up. So things like pgpool-ii don't even need a parser, it just knows it is a read only query because the protocol says so. JD -- Command Prompt, Inc. - http://www.commandprompt.com/ 509-416-6579 PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC, @cmdpromptinc For my dreams of your image that blossoms a rose in the deeps of my heart. - W.B. Yeats
On Tue, May 28, 2013 at 03:39:10PM -0700, Joshua D. Drake wrote: > > On 05/28/2013 03:36 PM, Bruce Momjian wrote: > > >>The other option would be to do it on query execute but that doesn't > >>seem as efficient as it would have to be parsed each time. Although > >>it would still be better than reading the actual SQL. > > > >Well, you could do SET TRANSACTION READ ONLY, and that would prevent any > >write transactions. You could assume it is a read query, and get the > >error and resubmit on the master if that happens, but that sounds > >inefficient. I thought you were asking for something where you could > >submit a query and it would report back as read/write or read-only. > > No I am suggesting something that before anything happens with the > parser, the protocol knows what is up. So things like pgpool-ii > don't even need a parser, it just knows it is a read only query > because the protocol says so. Oh, that is an interesting idea. The application is indicating it is read-only via the protocol, and poolers can optimize that. Don't we have the ability to pass arbitrary GUC values back through the protocol, e.g. transaction_read_only? If not, that might be a way to do this cleanly. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. +
On 05/28/2013 04:05 PM, Bruce Momjian wrote: > > On Tue, May 28, 2013 at 03:39:10PM -0700, Joshua D. Drake wrote: >> >> On 05/28/2013 03:36 PM, Bruce Momjian wrote: >> >>>> The other option would be to do it on query execute but that doesn't >>>> seem as efficient as it would have to be parsed each time. Although >>>> it would still be better than reading the actual SQL. >>> >>> Well, you could do SET TRANSACTION READ ONLY, and that would prevent any >>> write transactions. You could assume it is a read query, and get the >>> error and resubmit on the master if that happens, but that sounds >>> inefficient. I thought you were asking for something where you could >>> submit a query and it would report back as read/write or read-only. >> >> No I am suggesting something that before anything happens with the >> parser, the protocol knows what is up. So things like pgpool-ii >> don't even need a parser, it just knows it is a read only query >> because the protocol says so. > > Oh, that is an interesting idea. The application is indicating it is > read-only via the protocol, and poolers can optimize that. Don't we > have the ability to pass arbitrary GUC values back through the protocol, > e.g. transaction_read_only? If not, that might be a way to do this > cleanly. > I don't know but I don't think so. Anything that is calling SET is going to run through the parser. JD -- Command Prompt, Inc. - http://www.commandprompt.com/ 509-416-6579 PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC, @cmdpromptinc For my dreams of your image that blossoms a rose in the deeps of my heart. - W.B. Yeats
On 05/29/2013 05:11 AM, Bruce Momjian wrote: > Sure, it is on the TODO list: > > https://wiki.postgresql.org/wiki/Todo#.2Fcontrib.2Fpg_upgrade > > I can only get a link to pg_upgrade from there, so look two sections > below that for "Wire Protocol Changes". Thanks. The direct link is https://wiki.postgresql.org/wiki/Todo#Wire_Protocol_Changes for anyone looking for it later. -- Craig Ringer http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
On Tue, May 28, 2013 at 4:26 PM, Joshua D. Drake <jd@commandprompt.com> wrote: > > On 05/28/2013 02:18 PM, Bruce Momjian wrote: > >>> I would like to see the ability to define if a query is read only at >>> the protocol level, so that load balances that speak libpq can know >>> what to do with the query without parsing it. >> >> >> Sounds nice, but how would we do that? That would require libpq to know >> it, right? Do we pass anything back after parsing but before execution? >> Could it be optional? What about functions that modify the database >> --- isn't that only known at execution time? > > > I can't speak to the actual C code that would be required but from a user > space, I could see something like this: > > con = psycopg2.connect(database='testdb', user='test', transaction-type='r') > > Thus when the connection is made, before anything else is done, we know it > is a read only connection and therefore any load balancer speaking libpq > would also know it is a read only. The default of course would be r/w and > you would use a different connection handler for r/w or w queries. > you can do that today already, kind-of create an entry in pgbouncer that connect to host=read-only.servers.dns and make read-only.servers.dns to point to more than 1 ip. then when the application wants to do load balancing, just connect to the entry that points to read-only.servers.dns and let the magic happens which would be great is this to happen transparently to the application > The other option would be to do it on query execute but that doesn't seem as > efficient as it would have to be parsed each time. Although it would still > be better than reading the actual SQL. > another idea, as someone else mentioned, and i think has been discussed bedore is a function that says if the query is r-o or not... maybe even exporting the plan so we don't need to replan again... Not sure if that is possible, just hand waving... -- Jaime Casanova www.2ndQuadrant.com Professional PostgreSQL: Soporte 24x7 y capacitación Phone: +593 4 5107566 Cell: +593 987171157
> On 05/28/2013 04:05 PM, Bruce Momjian wrote: >> >> On Tue, May 28, 2013 at 03:39:10PM -0700, Joshua D. Drake wrote: >>> >>> On 05/28/2013 03:36 PM, Bruce Momjian wrote: >>> >>>>> The other option would be to do it on query execute but that doesn't >>>>> seem as efficient as it would have to be parsed each time. Although >>>>> it would still be better than reading the actual SQL. >>>> >>>> Well, you could do SET TRANSACTION READ ONLY, and that would prevent >>>> any >>>> write transactions. You could assume it is a read query, and get the >>>> error and resubmit on the master if that happens, but that sounds >>>> inefficient. I thought you were asking for something where you could >>>> submit a query and it would report back as read/write or read-only. >>> >>> No I am suggesting something that before anything happens with the >>> parser, the protocol knows what is up. So things like pgpool-ii >>> don't even need a parser, it just knows it is a read only query >>> because the protocol says so. >> >> Oh, that is an interesting idea. The application is indicating it is >> read-only via the protocol, and poolers can optimize that. Don't we >> have the ability to pass arbitrary GUC values back through the >> protocol, >> e.g. transaction_read_only? If not, that might be a way to do this >> cleanly. >> > > I don't know but I don't think so. Anything that is calling SET is > going to run through the parser. Right. SET command needs to be parsed by the parser. However, we already have embedded parameters in the start up packet, which needs to be recognized by pooler anyway. See "StartupMessage" section in: http://www.postgresql.org/docs/9.3/static/protocol-message-formats.html I am not sure backend currently permits to have default_transaction_read_only = on in the startup packet or not though. -- Tatsuo Ishii SRA OSS, Inc. Japan English: http://www.sraoss.co.jp/index_en.php Japanese: http://www.sraoss.co.jp
Bruce Momjian wrote: > On Mon, May 27, 2013 at 03:06:13PM -0400, Alvaro Herrera wrote: > > Bruce Momjian wrote: > > I would have each data segment be self-identifying, i.e. have a magic > > number at the beginning of the file and the relation OID, some fork > > identification and the segment number somewhere -- probably the special > > space of the first page. > > Is this something we want on the TODO? I was not clear how to do with > without making the first page format special or wasting space on all the > other pages. I don't think the special space has to necessarily be identically sized in all the pages -- I admit I haven't looked closely, but the special size is part of the page header. -- Álvaro Herrera http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
On Mon, May 27, 2013 at 9:41 AM, Simon Riggs <simon@2ndquadrant.com> wrote: > On 27 May 2013 15:36, Tom Lane <tgl@sss.pgh.pa.us> wrote: >> Bruce Momjian <bruce@momjian.us> writes: >>> On Mon, May 27, 2013 at 08:26:48AM -0400, Stephen Frost wrote: >>>> That said, many discussions and ideas do get shut down, perhaps too >>>> early, because of pg_upgrade considerations. If we had a plan to have >>>> an incompatible release in the future, those ideas and discussions might >>>> be able to progress to a point where we determine it's worth it to take >>>> the pain of a non-pg_upgrade-supported release. That's a bit of a >>>> stretch, in my view, but I suppose it's possible. Even so though, I >>>> would suggest that we put together a wiki page to list out those items >>>> and encourage people to add to such a list; perhaps having an item on >>>> that list would make discussion about it progress beyond "it breaks >>>> pg_upgrade". >> >>> Yes, we should be collecting things we want to do for a pg_upgrade break >>> so we can see the list all in one place. >> >> Precisely. We've said right along that we reserve the right to have a >> non-upgradable disk format change whenever sufficiently many reasons >> accumulate to do that. Here's one that's come up a few times: being able to tweak the out-of-line storage strategy, e.g. change the compression format used.I think some folks were lamenting the lack of a convenientbyte in the right place for that one.