Thread: pg_upgrade project status
The current project is not in good shape. Feature freeze is coming and nothing is committed. Currently there are three patches in the game: 1) Space reservation http://archives.postgresql.org/pgsql-hackers/2008-12/msg00886.php http://archives.postgresql.org/pgsql-hackers/2009-01/msg02031.php This patch is mandatory for page online conversion and MUST TO be part of postgreSQL 8.4. if not ... then we will be at the beginning next year. I sent updated version today. 2) pg_upgrade.sh http://archives.postgresql.org/pgsql-hackers/2008-12/msg00248.php Pg_upgrade.sh is shell script for catalog conversion. It works for 8.3->8.4 upgrade. It will be useful while we will not have better solution. Disadvantage is that it is korn shell script. The idea is to rewrite it in PERL which is more portable, but I'm not PERL expert and currently there is no workable solution. And what is deadline for it? We can delivery it later with 8.4.1, but best time for testing is during betas and RC period. 3) preupgrade script http://archives.postgresql.org/pgsql-hackers/2008-12/msg01273.php I sent WIP version of this script, which shows how reservation space feature will be used. This part is not important now. It will be important for 8.4->8.5 upgrade and cannot be finished until 8.5beta. Thats all Zdenek
Zdenek Kotala wrote: > The current project is not in good shape. Feature freeze is coming and > nothing is committed. Currently there are three patches in the game: > > [...] > 2) pg_upgrade.sh > http://archives.postgresql.org/pgsql-hackers/2008-12/msg00248.php > > Pg_upgrade.sh is shell script for catalog conversion. It works for > 8.3->8.4 upgrade. It will be useful while we will not have better > solution. Disadvantage is that it is korn shell script. The idea is to > rewrite it in PERL which is more portable, but I'm not PERL expert and > currently there is no workable solution. > > And what is deadline for it? We can delivery it later with 8.4.1, but > best time for testing is during betas and RC period. > > > I have had a very brief look at this. Translation to perl doesn't look difficult. I'll see what I can do during the next week or so. cheers andrew
Andrew Dunstan wrote: > Zdenek Kotala wrote: >> 2) pg_upgrade.sh >> http://archives.postgresql.org/pgsql-hackers/2008-12/msg00248.php >> >> Pg_upgrade.sh is shell script for catalog conversion. It works for >> 8.3->8.4 upgrade. It will be useful while we will not have better >> solution. Disadvantage is that it is korn shell script. The idea is to >> rewrite it in PERL which is more portable, but I'm not PERL expert and >> currently there is no workable solution. > > I have had a very brief look at this. Translation to perl doesn't look > difficult. I'll see what I can do during the next week or so. We don't require perl for any other feature, do we? Seems like a pretty onerous requireemnt for Windows in particular. We do use perl in the build scripts, but that's only required if you want to compile from source. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com
Heikki Linnakangas wrote: > Andrew Dunstan wrote: >> Zdenek Kotala wrote: >>> 2) pg_upgrade.sh >>> http://archives.postgresql.org/pgsql-hackers/2008-12/msg00248.php >>> >>> Pg_upgrade.sh is shell script for catalog conversion. It works for >>> 8.3->8.4 upgrade. It will be useful while we will not have better >>> solution. Disadvantage is that it is korn shell script. The idea is to >>> rewrite it in PERL which is more portable, but I'm not PERL expert and >>> currently there is no workable solution. >> >> I have had a very brief look at this. Translation to perl doesn't look >> difficult. I'll see what I can do during the next week or so. > > We don't require perl for any other feature, do we? Seems like a pretty > onerous requireemnt for Windows in particular. We do use perl in the > build scripts, but that's only required if you want to compile from > source. I think it's fairly easy to install Perl on Windows actually. It doesn't sound too onerous a requirement if you want in-place upgrade; actually it looks a very reasonable one. Much more reasonable than Korn shell in any case (or any shell for that matter; I think anything is going to be more of a potentially painful platform dependency than Perl). -- Alvaro Herrera http://www.CommandPrompt.com/ PostgreSQL Replication, Consulting, Custom Development, 24x7 support
On Tue, Jan 27, 2009 at 2:39 PM, Alvaro Herrera <alvherre@commandprompt.com> wrote: > I think it's fairly easy to install Perl on Windows actually. It > doesn't sound too onerous a requirement if you want in-place upgrade; > actually it looks a very reasonable one. There are installers for it, but given that we made a point of porting everything to C to avoid using any scripting languages on end-user machines when we ported to Windows, it seems strange to relax that 'policy' now for convenience. -- Dave Page EnterpriseDB UK: http://www.enterprisedb.com
On Tue, Jan 27, 2009 at 11:39:50AM -0300, Alvaro Herrera wrote: > Heikki Linnakangas wrote: > > Andrew Dunstan wrote: > >> Zdenek Kotala wrote: > >>> 2) pg_upgrade.sh > >>> http://archives.postgresql.org/pgsql-hackers/2008-12/msg00248.php > >>> > >>> Pg_upgrade.sh is shell script for catalog conversion. It works for > >>> 8.3->8.4 upgrade. It will be useful while we will not have better > >>> solution. Disadvantage is that it is korn shell script. The idea is to > >>> rewrite it in PERL which is more portable, but I'm not PERL expert and > >>> currently there is no workable solution. > >> > >> I have had a very brief look at this. Translation to perl doesn't look > >> difficult. I'll see what I can do during the next week or so. > > > > We don't require perl for any other feature, do we? Seems like a pretty > > onerous requireemnt for Windows in particular. We do use perl in the > > build scripts, but that's only required if you want to compile from > > source. > > I think it's fairly easy to install Perl on Windows actually. It > doesn't sound too onerous a requirement if you want in-place upgrade; > actually it looks a very reasonable one. > > Much more reasonable than Korn shell in any case (or any shell for that > matter; I think anything is going to be more of a potentially painful > platform dependency than Perl). > > -- > Alvaro Herrera http://www.CommandPrompt.com/ > PostgreSQL Replication, Consulting, Custom Development, 24x7 support > +1 I agree with Alvaro. Perl is a breeze to install on Windows with Activestate and that using shell code to perform this task adds a huge platform dependency to the code. Perl is a known and well defined quantity for scripting. Cheers, Ken
On Tue, Jan 27, 2009 at 8:44 AM, Zdenek Kotala <Zdenek.Kotala@sun.com> wrote: > The current project is not in good shape. Feature freeze is coming and > nothing is committed. Currently there are three patches in the game: Correction: feature freeze is long past. > 1) Space reservation > http://archives.postgresql.org/pgsql-hackers/2008-12/msg00886.php > http://archives.postgresql.org/pgsql-hackers/2009-01/msg02031.php > > This patch is mandatory for page online conversion and MUST TO be part > of postgreSQL 8.4. if not ... then we will be at the beginning next > year. > > I sent updated version today. I thought we pretty much had agreement that space reservation was not a good solution to anything, although I admit I'm not quite clear on what alternative was being proposed. > 2) pg_upgrade.sh > http://archives.postgresql.org/pgsql-hackers/2008-12/msg00248.php > > Pg_upgrade.sh is shell script for catalog conversion. It works for > 8.3->8.4 upgrade. It will be useful while we will not have better > solution. Disadvantage is that it is korn shell script. The idea is to > rewrite it in PERL which is more portable, but I'm not PERL expert and > currently there is no workable solution. > > And what is deadline for it? We can delivery it later with 8.4.1, but > best time for testing is during betas and RC period. Why is the deadline different than anything else? Surely the deadline is 11/1/2008, and you missed it. Admittedly, there is some window for reworking existing patches after the start of the commitfest, but this patch wasn't even added to the CommitFest wiki until December 5th, after being sent to the list the previous day. That's not close, and there's been little discussion of it on the mailing list since then, and given that it's written in ksh, it's clearly going to require a complete rewrite to be committable. Three months after the CommitFest started is not the right time to start a complete rewrite of a feature that wasn't even on time in the first place. There is nothing whatever to prevent you from releasing this on pgfoundry, but the idea that we should spend time on this rather than the half a dozen (or more) patches that have had far more work and are probably far closer to being committable strikes me as 100% wrong. > 3) preupgrade script > http://archives.postgresql.org/pgsql-hackers/2008-12/msg01273.php > > I sent WIP version of this script, which shows how reservation space > feature will be used. This part is not important now. It will be > important for 8.4->8.5 upgrade and cannot be finished until 8.5beta. ...Robert
Zdenek Kotala wrote: > 2) pg_upgrade.sh > http://archives.postgresql.org/pgsql-hackers/2008-12/msg00248.php > > Pg_upgrade.sh is shell script for catalog conversion. It works for > 8.3->8.4 upgrade. What's the reason this script uses a postmaster? It seems it would be easier to control if you used a standalone backend (--single) for each time you are piping stuff to psql. That would reduce the need to configure authentication, hostnames, etc etc. -- Alvaro Herrera http://www.CommandPrompt.com/ PostgreSQL Replication, Consulting, Custom Development, 24x7 support
On 1/27/09, Zdenek Kotala <Zdenek.Kotala@sun.com> wrote: > This patch is mandatory for page online conversion and MUST TO be part > of postgreSQL 8.4. if not ... then we will be at the beginning next > year. Just to clarify, does that mean that your patch has to be in for there to be any chance of in-place upgrade 8.4->8.5? merlin
Heikki Linnakangas wrote: > Andrew Dunstan wrote: >> Zdenek Kotala wrote: >>> 2) pg_upgrade.sh >>> http://archives.postgresql.org/pgsql-hackers/2008-12/msg00248.php >>> >>> Pg_upgrade.sh is shell script for catalog conversion. It works for >>> 8.3->8.4 upgrade. It will be useful while we will not have better >>> solution. Disadvantage is that it is korn shell script. The idea is to >>> rewrite it in PERL which is more portable, but I'm not PERL expert and >>> currently there is no workable solution. >> >> I have had a very brief look at this. Translation to perl doesn't >> look difficult. I'll see what I can do during the next week or so. > > We don't require perl for any other feature, do we? Seems like a > pretty onerous requireemnt for Windows in particular. We do use perl > in the build scripts, but that's only required if you want to compile > from source. Well, from that POV the only portable thing is to translate it into C. That's just a whole lot more work (remember initdb?). The perl port for Windows is easily installable, widely used and well regarded. It doesn't strike me as too high a price to pay for the ability to do upgrades, but I'll defer to more Windows-centric commenters. cheers andrew
Dave Page wrote: > On Tue, Jan 27, 2009 at 2:39 PM, Alvaro Herrera > <alvherre@commandprompt.com> wrote: > > > I think it's fairly easy to install Perl on Windows actually. It > > doesn't sound too onerous a requirement if you want in-place upgrade; > > actually it looks a very reasonable one. > > There are installers for it, but given that we made a point of porting > everything to C to avoid using any scripting languages on end-user > machines when we ported to Windows, it seems strange to relax that > 'policy' now for convenience. If you prefer to not have pg_upgrade at all for 8.4, feel free to request it to be written in C ... But I'm sure that's not what you meant. -- Alvaro Herrera http://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc.
Dave Page <dpage@pgadmin.org> writes:
> On Tue, Jan 27, 2009 at 2:39 PM, Alvaro Herrera
> <alvherre@commandprompt.com> wrote:
>> I think it's fairly easy to install Perl on Windows actually. It
>> doesn't sound too onerous a requirement if you want in-place upgrade;
>> actually it looks a very reasonable one.
> There are installers for it, but given that we made a point of porting
> everything to C to avoid using any scripting languages on end-user
> machines when we ported to Windows, it seems strange to relax that
> 'policy' now for convenience.
Indeed. We might put up with a perl script for awhile for the sake of
development expediency, but the long-term expectation would have to be
that someone would rewrite it in C. Given that, I wonder whether
there's much point in a rewrite into Perl if we already have a working
shell script. I suppose someone will say "but you'll get no testing
from Windows users then..."
regards, tom lane
Andrew Dunstan wrote: > > > Heikki Linnakangas wrote: >> Andrew Dunstan wrote: >>> Zdenek Kotala wrote: >>>> 2) pg_upgrade.sh >>>> http://archives.postgresql.org/pgsql-hackers/2008-12/msg00248.php >>>> >>>> Pg_upgrade.sh is shell script for catalog conversion. It works for >>>> 8.3->8.4 upgrade. It will be useful while we will not have better >>>> solution. Disadvantage is that it is korn shell script. The idea is to >>>> rewrite it in PERL which is more portable, but I'm not PERL expert and >>>> currently there is no workable solution. >>> >>> I have had a very brief look at this. Translation to perl doesn't >>> look difficult. I'll see what I can do during the next week or so. >> >> We don't require perl for any other feature, do we? Seems like a >> pretty onerous requireemnt for Windows in particular. We do use perl >> in the build scripts, but that's only required if you want to compile >> from source. > > Well, from that POV the only portable thing is to translate it into C. > That's just a whole lot more work (remember initdb?). The perl port for > Windows is easily installable, widely used and well regarded. It doesn't > strike me as too high a price to pay for the ability to do upgrades, but > I'll defer to more Windows-centric commenters. Either way, there's no point to discuss that in detail until there actually is a working implementation out there... perl will do fine until then. Once we have that, we can discuss if doing it in C will be worthwhile, or if we're just going to require perl for that one feature. I have a hard time thinking that we'll have wasted a lot of time on first doing a perl implementation if we have to rewrite it in C later. The other way around would be a waste though. The amount of time spent on the perl implementation I expect to be a *lot* less than the combination of thinking up the *way* to do it in general and the C implementation time. //Magnus
On Tue, Jan 27, 2009 at 2:49 PM, Alvaro Herrera <alvherre@commandprompt.com> wrote: > Dave Page wrote: >> On Tue, Jan 27, 2009 at 2:39 PM, Alvaro Herrera >> <alvherre@commandprompt.com> wrote: >> >> > I think it's fairly easy to install Perl on Windows actually. It >> > doesn't sound too onerous a requirement if you want in-place upgrade; >> > actually it looks a very reasonable one. >> >> There are installers for it, but given that we made a point of porting >> everything to C to avoid using any scripting languages on end-user >> machines when we ported to Windows, it seems strange to relax that >> 'policy' now for convenience. > > If you prefer to not have pg_upgrade at all for 8.4, feel free to > request it to be written in C ... But I'm sure that's not what you > meant. I'd rather it was written in an appropriate language before feature freeze, not in a language that makes it easier for the author but a shade harder for thousands of users three months into feature freeze. -- Dave Page EnterpriseDB UK: http://www.enterprisedb.com
* Magnus Hagander (magnus@hagander.net) wrote: > Either way, there's no point to discuss that in detail until there > actually is a working implementation out there... perl will do fine > until then. Once we have that, we can discuss if doing it in C will be > worthwhile, or if we're just going to require perl for that one feature. +1 Stephen
* Robert Haas (robertmhaas@gmail.com) wrote:
[pg_upgrade...]
> Why is the deadline different than anything else?
err, isn't it because it'd be kind of difficult to do an upgrade script
with large catalog-changing patches outstanding..? I thought some
leeway was given for pg_upgrade specifically due to that, tho perhaps
I'm wrong.
Thanks,
Stephen
On Tue, Jan 27, 2009 at 10:08 AM, Stephen Frost <sfrost@snowman.net> wrote: > * Robert Haas (robertmhaas@gmail.com) wrote: > [pg_upgrade...] >> Why is the deadline different than anything else? > > err, isn't it because it'd be kind of difficult to do an upgrade script > with large catalog-changing patches outstanding..? I thought some > leeway was given for pg_upgrade specifically due to that, tho perhaps > I'm wrong. Sure... if this script had been 100% commitable on 11/1 and now needed to be adjusted, I can't imagine anyone objecting. But the patch wasn't submitted until 12/4 and still needs a complete rewrite in a different programming language as of 1/27. Do you think we would be arguing about whether to accept Hot Standby now if it were written in ksh? And that was at least submitted on time. ...Robert
> I think it's fairly easy to install Perl on Windows actually. It > doesn't sound too onerous a requirement if you want in-place upgrade; > actually it looks a very reasonable one. > > Much more reasonable than Korn shell in any case (or any shell for that > matter; I think anything is going to be more of a potentially painful > platform dependency than Perl). > May I humbly recommend to rewrite in Python? That should be as difficult / easy as PERL, AND there is a very robust py2exe implementation, which allows to create a single .exe file which contains everything. Python is present on all Linux, Windows users are totally comfortable with .exe files. Harald -- GHUM Harald Massa persuadere et programmare Harald Armin Massa Spielberger Straße 49 70435 Stuttgart 0173/9409607 no fx, no carrier pigeon - EuroPython 2009 will take place in Birmingham - Stay tuned!
Hi > I have had a very brief look at this. Translation to perl doesn't look > difficult. I'll see what I can do during the next week or so. Perhaps I can lend you a hand if you need help with this. -- Med venlig hilsen Kaare Rasmussen, Jasonic Jasonic Telefon: +45 3816 2582 Nordre Fasanvej 12 2000 Frederiksberg Email: kaare@jasonic.dk
Harald Armin Massa wrote: >> I think it's fairly easy to install Perl on Windows actually. It >> doesn't sound too onerous a requirement if you want in-place upgrade; >> actually it looks a very reasonable one. >> >> Much more reasonable than Korn shell in any case (or any shell for that >> matter; I think anything is going to be more of a potentially painful >> platform dependency than Perl). >> >> > > May I humbly recommend to rewrite in Python? That should be as > difficult / easy as PERL, AND there is a very robust py2exe > implementation, which allows to create a single .exe file which > contains everything. > > Python is present on all Linux, Windows users are totally comfortable > with .exe files. > > > No, I don't think so ;-) Without getting into language wars, Perl is already our de facto cross-platform scripting tool. We don't need to be adding extra knowledge requirements to the project, nor extra build requirements (right now, perl is already required for building from source, or when building with MSVC, or when running a buildfarm animal) cheers andrew
Alvaro Herrera píše v út 27. 01. 2009 v 11:47 -0300: > Zdenek Kotala wrote: > > > 2) pg_upgrade.sh > > http://archives.postgresql.org/pgsql-hackers/2008-12/msg00248.php > > > > Pg_upgrade.sh is shell script for catalog conversion. It works for > > 8.3->8.4 upgrade. > > What's the reason this script uses a postmaster? It seems it would be > easier to control if you used a standalone backend (--single) for each > time you are piping stuff to psql. That would reduce the need to > configure authentication, hostnames, etc etc. It is 8.1->8.2 upgrade relict. IIRC 8.1 does not have --single. I think single mode can be used. thanks Zdenek
On Tue, 2009-01-27 at 09:48 -0500, Andrew Dunstan wrote: > > We don't require perl for any other feature, do we? Seems like a > > pretty onerous requireemnt for Windows in particular. We do use perl > > in the build scripts, but that's only required if you want to compile > > from source. > > Well, from that POV the only portable thing is to translate it into C. > That's just a whole lot more work (remember initdb?). The perl port for > Windows is easily installable, widely used and well regarded. It doesn't > strike me as too high a price to pay for the ability to do upgrades, but > I'll defer to more Windows-centric commenters. Actually as much as perl is ubiquitous it isn't. What version of perl shall we require? Will we require other modules? Does that version work on all our supported platforms (HPUX, NETBSD?) Joshua D. Drake -- PostgreSQL - XMPP: jdrake@jabber.postgresql.org Consulting, Development, Support, Training 503-667-4564 - http://www.commandprompt.com/ The PostgreSQL Company, serving since 1997
Joshua D. Drake wrote: > On Tue, 2009-01-27 at 09:48 -0500, Andrew Dunstan wrote: > > >>> We don't require perl for any other feature, do we? Seems like a >>> pretty onerous requireemnt for Windows in particular. We do use perl >>> in the build scripts, but that's only required if you want to compile >>> from source. >>> >> Well, from that POV the only portable thing is to translate it into C. >> That's just a whole lot more work (remember initdb?). The perl port for >> Windows is easily installable, widely used and well regarded. It doesn't >> strike me as too high a price to pay for the ability to do upgrades, but >> I'll defer to more Windows-centric commenters. >> > > Actually as much as perl is ubiquitous it isn't. What version of perl > shall we require? Will we require other modules? Does that version work > on all our supported platforms (HPUX, NETBSD?) > > > That's what my brief examination of the script was about - looking to see if it could be translated portably. I think it very probably can. I suspect it won't need any modules at all. I suspect any cheers andrew
On Tue, 2009-01-27 at 10:56 -0500, Andrew Dunstan wrote: > > > That's what my brief examination of the script was about - looking to > see if it could be translated portably. I think it very probably can. I > suspect it won't need any modules at all. I suspect any Accidental ctrl-enter? O.k. well you certainly know more about perl than I do. If you are comfortable with the portability, I have no complaints (except that I am not willing to wait for it to be done for 8.4). Joshua D. Drake > > cheers > > andrew > -- PostgreSQL - XMPP: jdrake@jabber.postgresql.org Consulting, Development, Support, Training 503-667-4564 - http://www.commandprompt.com/ The PostgreSQL Company, serving since 1997
On Tue, Jan 27, 2009 at 04:23:18PM +0100, Harald Armin Massa wrote: > > I think it's fairly easy to install Perl on Windows actually. It > > doesn't sound too onerous a requirement if you want in-place upgrade; > > actually it looks a very reasonable one. > > > > Much more reasonable than Korn shell in any case (or any shell for that > > matter; I think anything is going to be more of a potentially painful > > platform dependency than Perl). > > > > May I humbly recommend to rewrite in Python? That should be as > difficult / easy as PERL, AND there is a very robust py2exe > implementation, which allows to create a single .exe file which > contains everything. > > Python is present on all Linux, Windows users are totally comfortable > with .exe files. > > Harald > > Great idea, perl2exe is available as well and will allow the continued use of perl for our internal scripting language. Cheers, Ken
Merlin Moncure píše v út 27. 01. 2009 v 09:47 -0500:
> On 1/27/09, Zdenek Kotala <Zdenek.Kotala@sun.com> wrote:
> > This patch is mandatory for page online conversion and MUST TO be part
> > of postgreSQL 8.4. if not ... then we will be at the beginning next
> > year.
>
> Just to clarify, does that mean that your patch has to be in for there
> to be any chance of in-place upgrade 8.4->8.5?
Ok, There two patch in the queue for 8.5 which will bump page layout
version. Then we will need it.
Zdenek
Robert Haas píše v út 27. 01. 2009 v 09:45 -0500: > > > 1) Space reservation > > http://archives.postgresql.org/pgsql-hackers/2008-12/msg00886.php > > http://archives.postgresql.org/pgsql-hackers/2009-01/msg02031.php > > > > This patch is mandatory for page online conversion and MUST TO be part > > of postgreSQL 8.4. if not ... then we will be at the beginning next > > year. > > > > I sent updated version today. > > I thought we pretty much had agreement that space reservation was not > a good solution to anything, although I admit I'm not quite clear on > what alternative was being proposed. Maybe I miss something, but space reservation was selected as a best way. Please, Could you point me related mailing thread? Zdenek
Magnus Hagander píše v út 27. 01. 2009 v 15:54 +0100:
>
> I have a hard time thinking that we'll have wasted a lot of time on
> first doing a perl implementation if we have to rewrite it in C later.
> The other way around would be a waste though. The amount of time spent
> on the perl implementation I expect to be a *lot* less than the
> combination of thinking up the *way* to do it in general and the C
> implementation time.
Agree with Magnus. I'm happy with korn shell for now :-). And we need
better solution anyway for future releases. Current approach does not
support e.g. dropped column.
Zdenek
Dave Page píše v út 27. 01. 2009 v 14:56 +0000: > On Tue, Jan 27, 2009 at 2:49 PM, Alvaro Herrera > <alvherre@commandprompt.com> wrote: > > Dave Page wrote: > >> On Tue, Jan 27, 2009 at 2:39 PM, Alvaro Herrera > >> <alvherre@commandprompt.com> wrote: > >> > >> > I think it's fairly easy to install Perl on Windows actually. It > >> > doesn't sound too onerous a requirement if you want in-place upgrade; > >> > actually it looks a very reasonable one. > >> > >> There are installers for it, but given that we made a point of porting > >> everything to C to avoid using any scripting languages on end-user > >> machines when we ported to Windows, it seems strange to relax that > >> 'policy' now for convenience. > > > > If you prefer to not have pg_upgrade at all for 8.4, feel free to > > request it to be written in C ... But I'm sure that's not what you > > meant. > > I'd rather it was written in an appropriate language before feature > freeze, not in a language that makes it easier for the author but a > shade harder for thousands of users three months into feature freeze. The script is one year old. It is integrated in the OpenSolaris and Solaris 10U6. I presented the script on PGCon2008. Everybody could complain about it long time ago. And from my point of view it is first step then better solution will be developed in C - (see my presentation - "initdb upgrade" command). Zdenek
Zdenek Kotala wrote: > Robert Haas píše v út 27. 01. 2009 v 09:45 -0500: > >>> 1) Space reservation >>> http://archives.postgresql.org/pgsql-hackers/2008-12/msg00886.php >>> http://archives.postgresql.org/pgsql-hackers/2009-01/msg02031.php >>> >>> This patch is mandatory for page online conversion and MUST TO be part >>> of postgreSQL 8.4. if not ... then we will be at the beginning next >>> year. >>> >>> I sent updated version today. >> I thought we pretty much had agreement that space reservation was not >> a good solution to anything, although I admit I'm not quite clear on >> what alternative was being proposed. > > Maybe I miss something, but space reservation was selected as a best > way. Please, Could you point me related mailing thread? Space reservation is the way to go, but I remain of the opinion that trying to predict the future is futile. When we know what we need (= around the beginning of 8.5 beta), we can backpatch a patch to reserve the needed amount of space on pages. It has to be dead simple to consider applying it to a stable branch, but something like "reserve X bytes for heap pages and Y bytes for b-tree pages if pre_upgrade_mode GUC is set" should be OK. And we don't want to do anything more complicated than that anyway before we know what we need. (http://archives.postgresql.org/message-id/49425D07.6030701@enterprisedb.com) -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com
Zdenek Kotala wrote: > Magnus Hagander píše v út 27. 01. 2009 v 15:54 +0100: > >> I have a hard time thinking that we'll have wasted a lot of time on >> first doing a perl implementation if we have to rewrite it in C later. >> The other way around would be a waste though. The amount of time spent >> on the perl implementation I expect to be a *lot* less than the >> combination of thinking up the *way* to do it in general and the C >> implementation time. > > Agree with Magnus. I'm happy with korn shell for now :-). And we need > better solution anyway for future releases. Current approach does not > support e.g. dropped column. FWIW, pg-migrator is written in C, and last I looked it's roughly on par feature-wise. Perhaps Zdenek can list what exactly the differences are, so that we can figure out how much effort it would be to just fill the gaps in pg-migrator. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com
Heikki Linnakangas píše v út 27. 01. 2009 v 21:07 +0200: > Zdenek Kotala wrote: > > Robert Haas píše v út 27. 01. 2009 v 09:45 -0500: > > > >>> 1) Space reservation > >>> http://archives.postgresql.org/pgsql-hackers/2008-12/msg00886.php > >>> http://archives.postgresql.org/pgsql-hackers/2009-01/msg02031.php > >>> > >>> This patch is mandatory for page online conversion and MUST TO be part > >>> of postgreSQL 8.4. if not ... then we will be at the beginning next > >>> year. > >>> > >>> I sent updated version today. > >> I thought we pretty much had agreement that space reservation was not > >> a good solution to anything, although I admit I'm not quite clear on > >> what alternative was being proposed. > > > > Maybe I miss something, but space reservation was selected as a best > > way. Please, Could you point me related mailing thread? > > Space reservation is the way to go, but I remain of the opinion that > trying to predict the future is futile. When we know what we need (= > around the beginning of 8.5 beta), we can backpatch a patch to reserve > the needed amount of space on pages. It has to be dead simple to > consider applying it to a stable branch, but something like "reserve X > bytes for heap pages and Y bytes for b-tree pages if pre_upgrade_mode > GUC is set" should be OK. And we don't want to do anything more > complicated than that anyway before we know what we need. Set general value for heap or btree is possible but not optimal. For example If you have 5TB table and page layout changes do not affected this table but some small table needs perform a cleanup then 5TB will be processed in general approach, but in optimal variant it will be mark as prepared on upgrade automatically. The patch which I send do this. It is optimal. Only what we don't know is correct calculation, but when 8.5 will be release we will able to create script/procedure which calculate exact space for each relation. Nothing should be backpatched, because script calls something like ALTER TABLE test SET (rs_perpage=4); or UPDATE pg_class SET rs_ perpage=4 where relname='test'; This is a whole idea. thanks Zdenek
Zdenek Kotala <Zdenek.Kotala@Sun.COM> writes:
> Merlin Moncure píše v út 27. 01. 2009 v 09:47 -0500:
>> Just to clarify, does that mean that your patch has to be in for there
>> to be any chance of in-place upgrade 8.4->8.5?
> Ok, There two patch in the queue for 8.5 which will bump page layout
> version. Then we will need it.
I see nothing on the 2009-First list that requires any such thing.
In any case we've been over this ground before: we have agreed in the
past that we'd be willing to reject/postpone patches that change on-disk
data layout if in-place upgrade would otherwise be available. I think
that space reservation is extremely far down the list of "must haves"
for getting 8.4->8.5 upgrade into place. You should first work on an
update process that supports catalog changes, and get that committed.
Once that's in place you'll have enough political capital to prevent
changes in user data layout, and then we'd start to think about schemes
like space reservation that could relax that ground rule.
regards, tom lane
On Tue, Jan 27, 2009 at 7:04 PM, Zdenek Kotala <Zdenek.Kotala@sun.com> wrote: > > Dave Page píše v út 27. 01. 2009 v 14:56 +0000: >> I'd rather it was written in an appropriate language before feature >> freeze, not in a language that makes it easier for the author but a >> shade harder for thousands of users three months into feature freeze. > > The script is one year old. It is integrated in the OpenSolaris and > Solaris 10U6. I presented the script on PGCon2008. Everybody could > complain about it long time ago. Neither of which are the correct place to submit work for inclusion in PostgreSQL. We're currently arguing about bouncing features that are in late stages of development and were submitted long before feature freeze. Your upgrade script is in the early stages of development (although it may be mature as a korn script), and needs to be entirely rewritten in a different language before it can even be considered for inclusion. If it's been around for a year, why hasn't it been submitted long ago so we could have rewritten and reviewed etc. in plenty of time? I'd love to see UIP, but even I, as someone that generally (and sometimes loudly) supports good work that is submitted close to feature freeze am having a hard time seeing why this work should be accepted so late, even if its in /contrib. -- Dave Page EnterpriseDB UK: http://www.enterprisedb.com
Zdenek Kotala wrote: > Set general value for heap or btree is possible but not optimal. For > example If you have 5TB table and page layout changes do not affected > this table but some small table needs perform a cleanup then 5TB will be > processed in general approach, but in optimal variant it will be mark as > prepared on upgrade automatically. Only the space reservation of *new* inserts/updates need to be integrated in the backend. In addition to that, we'll need a pre-upgrade script that decides which tables need processing, and process them. This is no different whether we accept some generic space reservation patch now, or backpatch a more precise one later. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com
Heikki Linnakangas píše v út 27. 01. 2009 v 21:13 +0200: > Zdenek Kotala wrote: > > Magnus Hagander píše v út 27. 01. 2009 v 15:54 +0100: > > > >> I have a hard time thinking that we'll have wasted a lot of time on > >> first doing a perl implementation if we have to rewrite it in C later. > >> The other way around would be a waste though. The amount of time spent > >> on the perl implementation I expect to be a *lot* less than the > >> combination of thinking up the *way* to do it in general and the C > >> implementation time. > > > > Agree with Magnus. I'm happy with korn shell for now :-). And we need > > better solution anyway for future releases. Current approach does not > > support e.g. dropped column. > > FWIW, pg-migrator is written in C, and last I looked it's roughly on par > feature-wise. Perhaps Zdenek can list what exactly the differences are, > so that we can figure out how much effort it would be to just fill the > gaps in pg-migrator. The reason why I wrote own script was that I found lot of small issues which needed to solve. OK finally maybe fixing them could take short time, but I did not thing this one year ago :(. OK there is what I remember: The main difference is interaction with postgreSQL - stoping/starting and so on. IIRC pg migrator does not do that. pg_upgrade script sets trust permisions on database to perform upgrade and so on. The migration process is mostly identical. if IIRC pg_migrator ignores locales - it fails when is not run under English locales and also initdb is not called with correct parameters. Check also freezing database. I think it is not correctly performed. Remove all ANSI color staff. You see nothing when you run it on black background :(. I guess, Tablespaces are handled differently. I try to keep data on same mounpoint. And last I think license is not BSD, I don't know how big problem it is. Zdenek
Tom Lane píše v út 27. 01. 2009 v 14:31 -0500: > Zdenek Kotala <Zdenek.Kotala@Sun.COM> writes: > > Merlin Moncure píše v út 27. 01. 2009 v 09:47 -0500: > >> Just to clarify, does that mean that your patch has to be in for there > >> to be any chance of in-place upgrade 8.4->8.5? > > > Ok, There two patch in the queue for 8.5 which will bump page layout > > version. Then we will need it. > > I see nothing on the 2009-First list that requires any such thing. There are page CRC and item alignment optimization. And If IIRC that somebody what to modify compression? > In any case we've been over this ground before: we have agreed in the > past that we'd be willing to reject/postpone patches that change on-disk > data layout if in-place upgrade would otherwise be available. I think > that space reservation is extremely far down the list of "must haves" > for getting 8.4->8.5 upgrade into place. If it means that we will not change Page Layout Version in 8.5 then I'm happy. > You should first work on an > update process that supports catalog changes, and get that committed. > Once that's in place you'll have enough political capital to prevent > changes in user data layout, and then we'd start to think about schemes > like space reservation that could relax that ground rule. OK. There is pg_upgrade.sh for 8.3->8.4 while I or someone develop something better. But new solution will be at first for 8.4->8.5. thanks Zdenek
Heikki Linnakangas píše v út 27. 01. 2009 v 21:37 +0200: > Zdenek Kotala wrote: > > Set general value for heap or btree is possible but not optimal. For > > example If you have 5TB table and page layout changes do not affected > > this table but some small table needs perform a cleanup then 5TB will be > > processed in general approach, but in optimal variant it will be mark as > > prepared on upgrade automatically. > > Only the space reservation of *new* inserts/updates need to be > integrated in the backend. In addition to that, we'll need a pre-upgrade > script that decides which tables need processing, and process them. This > is no different whether we accept some generic space reservation patch > now, or backpatch a more precise one later. If it is possible then OK, but I little bit afraid about it. Zdenek
On Tue, 2009-01-27 at 20:46 +0100, Zdenek Kotala wrote: > And last I think license is not BSD, I don't know how big problem it is. Uhh.. that kills it. We only accept BSD. Joshua D. Drake > > Zdenek > > > > -- PostgreSQL - XMPP: jdrake@jabber.postgresql.org Consulting, Development, Support, Training 503-667-4564 - http://www.commandprompt.com/ The PostgreSQL Company, serving since 1997
Joshua D. Drake wrote: > On Tue, 2009-01-27 at 20:46 +0100, Zdenek Kotala wrote: > >> And last I think license is not BSD, I don't know how big problem it is. > > Uhh.. that kills it. We only accept BSD. Actually, that was just fixed. Once somebody raised the issue, EDB changed the license (there have been on other contributors). I'm waiting for some documentation from Denis on that, but I expect to have it in a few days. --Josh
Joshua D. Drake wrote: > On Tue, 2009-01-27 at 20:46 +0100, Zdenek Kotala wrote: > >> And last I think license is not BSD, I don't know how big problem it is. > > Uhh.. that kills it. We only accept BSD. Actually, it has been relicensed to BSD license, after a short discussion started by Bruce within EnterpriseDB (all the code is written by EDB employees). This happened yesterday, so I can't blame you for not noticing ;-). -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com
On Tue, 2009-01-27 at 22:12 +0200, Heikki Linnakangas wrote: > Joshua D. Drake wrote: > > On Tue, 2009-01-27 at 20:46 +0100, Zdenek Kotala wrote: > > > >> And last I think license is not BSD, I don't know how big problem it is. > > > > Uhh.. that kills it. We only accept BSD. > > Actually, it has been relicensed to BSD license, after a short > discussion started by Bruce within EnterpriseDB (all the code is written > by EDB employees). This happened yesterday, so I can't blame you for not > noticing ;-). Phew! :) Joshua D. Drake > > -- > Heikki Linnakangas > EnterpriseDB http://www.enterprisedb.com > -- PostgreSQL - XMPP: jdrake@jabber.postgresql.org Consulting, Development, Support, Training 503-667-4564 - http://www.commandprompt.com/ The PostgreSQL Company, serving since 1997
Dave Page píše v út 27. 01. 2009 v 19:36 +0000: > On Tue, Jan 27, 2009 at 7:04 PM, Zdenek Kotala <Zdenek.Kotala@sun.com> wrote: > > > > Dave Page píše v út 27. 01. 2009 v 14:56 +0000: > >> I'd rather it was written in an appropriate language before feature > >> freeze, not in a language that makes it easier for the author but a > >> shade harder for thousands of users three months into feature freeze. > > > > The script is one year old. It is integrated in the OpenSolaris and > > Solaris 10U6. I presented the script on PGCon2008. Everybody could > > complain about it long time ago. > > Neither of which are the correct place to submit work for inclusion in > PostgreSQL. http://archives.postgresql.org/pgsql-hackers/2008-04/msg00990.php I sent it in April. Zdenek
On Tue, Jan 27, 2009 at 8:31 PM, Zdenek Kotala <Zdenek.Kotala@sun.com> wrote: > > Dave Page píše v út 27. 01. 2009 v 19:36 +0000: >> On Tue, Jan 27, 2009 at 7:04 PM, Zdenek Kotala <Zdenek.Kotala@sun.com> wrote: >> > >> > Dave Page píše v út 27. 01. 2009 v 14:56 +0000: >> >> I'd rather it was written in an appropriate language before feature >> >> freeze, not in a language that makes it easier for the author but a >> >> shade harder for thousands of users three months into feature freeze. >> > >> > The script is one year old. It is integrated in the OpenSolaris and >> > Solaris 10U6. I presented the script on PGCon2008. Everybody could >> > complain about it long time ago. >> >> Neither of which are the correct place to submit work for inclusion in >> PostgreSQL. > > http://archives.postgresql.org/pgsql-hackers/2008-04/msg00990.php > > I sent it in April. Thats a patch to the server, not a korn script. Wrong URL perhaps? -- Dave Page EnterpriseDB UK: http://www.enterprisedb.com
Dave Page píše v út 27. 01. 2009 v 20:36 +0000: > On Tue, Jan 27, 2009 at 8:31 PM, Zdenek Kotala <Zdenek.Kotala@sun.com> wrote: > >> > >> Neither of which are the correct place to submit work for inclusion in > >> PostgreSQL. > > > > http://archives.postgresql.org/pgsql-hackers/2008-04/msg00990.php > > > > I sent it in April. > > Thats a patch to the server, not a korn script. Wrong URL perhaps? Sorry, I copied wrong email URL from thread http://archives.postgresql.org/pgsql-hackers/2008-04/msg01051.php But you can read there that it is korn shell script and there is link to it. Thats all Zdenek
On Tuesday 27 January 2009 21:36:01 Dave Page wrote: > If it's been around for a year, why hasn't it been > submitted long ago so we could have rewritten and reviewed etc. in > plenty of time? It's in the commit fest listing, and the discussion about which programming language to port it to happened weeks ago.
On Tue, Jan 27, 2009 at 8:52 PM, Zdenek Kotala <Zdenek.Kotala@sun.com> wrote: > > http://archives.postgresql.org/pgsql-hackers/2008-04/msg01051.php > > But you can read there that it is korn shell script and there is link to > it. OK, yes that does mention it. It's not exactly clear that it's something you wanted to include though. -- Dave Page EnterpriseDB UK: http://www.enterprisedb.com
On Tuesday 27 January 2009 16:52:15 Tom Lane wrote: > Indeed. We might put up with a perl script for awhile for the sake of > development expediency, but the long-term expectation would have to be > that someone would rewrite it in C. Given that, I wonder whether > there's much point in a rewrite into Perl if we already have a working > shell script. I suppose someone will say "but you'll get no testing > from Windows users then..." The existing ksh script needs about two weeks of work to make it work outside of Solaris and to make it more robust. Then you might as well rewrite it in a more portable and robust language. But yes, you have to think weeks here in terms of getting this polished.
On Tuesday 27 January 2009 16:54:02 Magnus Hagander wrote: > Either way, there's no point to discuss that in detail until there > actually is a working implementation out there... perl will do fine > until then There is a working implementation out there. It is -- you guessed it -- sitting in the commit fest queue waiting for review.
On Tue, Jan 27, 2009 at 8:54 PM, Peter Eisentraut <peter_e@gmx.net> wrote: > On Tuesday 27 January 2009 21:36:01 Dave Page wrote: >> If it's been around for a year, why hasn't it been >> submitted long ago so we could have rewritten and reviewed etc. in >> plenty of time? > > It's in the commit fest listing, and the discussion about which programming > language to port it to happened weeks ago. It was added to the commitfest page on December 5th, and links to an email from the day before. The only discussion on the language I can see in the linked thread is David Fetter offering to rewrite the code in perl. -- Dave Page EnterpriseDB UK: http://www.enterprisedb.com
On Tuesday 27 January 2009 17:27:32 Kaare Rasmussen wrote: > Hi > > > I have had a very brief look at this. Translation to perl doesn't look > > difficult. I'll see what I can do during the next week or so. > > Perhaps I can lend you a hand if you need help with this. Not to stop you are anything, but in the lifetime of this script, there have been about half a dozen people offering to port or already porting it to Perl. What it needs is a design and functional review to see if it is worth porting or doing anything with at all.
On Tue, 27 Jan 2009, Peter Eisentraut wrote: > On Tuesday 27 January 2009 21:36:01 Dave Page wrote: >> If it's been around for a year, why hasn't it been >> submitted long ago so we could have rewritten and reviewed etc. in >> plenty of time? > > It's in the commit fest listing, and the discussion about which programming > language to port it to happened weeks ago. But as Robert Haas pointed out upthread that didn't show up there until after the start of the main commitfest, and I will admit being the culprit who snuck it onto there. The script was brought up repeatedly going back to April, but Zdenek didn't explicitly submit it as part of his final set for the commitfest. It was obvious to me you needed it all along--Zdenek's other CF submissions implicitly depended on it if your goal was to upgrade from 8.3 to 8.4 in-place--which is why I nudged him to do that and then added the result to the page. I'm not sure what arguing over the exact sequence here accomplishes anyway. It's come up a bunch of times before, nobody ran with it, and Zdenek was more concerned about space reservation. Anyway, I see multiple people who seem excited by playing with a Perl port. I found a shell->Perl conversion tool recently: http://search.cpan.org/~clive/App-sh2p-0.04/ If you comment out all the calls to create_mapping which it chokes on for some reason, it produces a non-functional but helpful conversion into Perl. But since the primary dirty job here is to review whether the original script works as expected on the database problems it tries to solve rather than play with programming languages, I didn't think reporting progress just on that front was particularly exciting. -- * Greg Smith gsmith@gregsmith.com http://www.gregsmith.com Baltimore, MD
On Tuesday 27 January 2009 23:02:30 Dave Page wrote: > On Tue, Jan 27, 2009 at 8:54 PM, Peter Eisentraut <peter_e@gmx.net> wrote: > > On Tuesday 27 January 2009 21:36:01 Dave Page wrote: > >> If it's been around for a year, why hasn't it been > >> submitted long ago so we could have rewritten and reviewed etc. in > >> plenty of time? > > > > It's in the commit fest listing, and the discussion about which > > programming language to port it to happened weeks ago. > > It was added to the commitfest page on December 5th, and links to an > email from the day before. The only discussion on the language I can > see in the linked thread is David Fetter offering to rewrite the code > in perl. Well, the is ultimately about the same discussion we are having now, except that people are a lot more excited when you threaten to cancel the project. ;-)
Peter Eisentraut <peter_e@gmx.net> writes:
> On Tuesday 27 January 2009 16:52:15 Tom Lane wrote:
>> Indeed. We might put up with a perl script for awhile for the sake of
>> development expediency, but the long-term expectation would have to be
>> that someone would rewrite it in C. Given that, I wonder whether
>> there's much point in a rewrite into Perl if we already have a working
>> shell script. I suppose someone will say "but you'll get no testing
>> from Windows users then..."
> The existing ksh script needs about two weeks of work to make it work outside
> of Solaris and to make it more robust. Then you might as well rewrite it in
> a more portable and robust language.
Agreed, if it has to be gone over in that much detail, conversion to perl
might not be a bad idea. I still say it'd have to be C eventually, but
it'd be good to use something more concise until all the design issues
are shaken out.
regards, tom lane
>> I thought we pretty much had agreement that space reservation was not >> a good solution to anything, although I admit I'm not quite clear on >> what alternative was being proposed. > > Maybe I miss something, but space reservation was selected as a best > way. Please, Could you point me related mailing thread? Hmm, maybe I misrepresented it slightly. But I did find these: http://archives.postgresql.org/message-id/49425D07.6030701@enterprisedb.com http://archives.postgresql.org/message-id/4948D567.4060908@enterprisedb.com ...and just today: http://archives.postgresql.org/message-id/497F5B7D.90904@enterprisedb.com ...which I think are at least an argument against this patch, if not a total refutation of the concept of space reservation in general. Personally, I think you're both barking up the wrong tree. If the amount of space that we need to reserve is more complicated than X bytes per page or per tuple, then your approach will fail. But it doesn't have to be a lot more complicated than that before Heikki's approach of back-patching the space reservation code will fail too. I foresee vehement arguments about whether the space reservation code for feature Y is too complex to back-patch, with the argument going something like this: UIP Dude: If we don't back-patch this code, we won't be able to upgrade-in-place for this release. That's a huge problem, man! Conservative Guy: Yeah, but if we do, and we fry somebody's database, that's a lot worse than not being able to upgrade-in-place. UIP Dude: I don't think that's going to happen. This code is pretty solid. Conservative Guy: You can't prove it doesn't have bugs. Conservative Guy has a point, but the real issue is that UIP Dude is making extra work for himself by splitting the work of upgrading the system between release N and release N+1. Sure, with enough work, it will be possible to test the back branch enough to convince yourself that there are no horrible bugs, and it will also be possible to test that CVS HEAD can complete the upgrade, but now you have to exhaustively test TWO releases both independently and for compatibility with each other. It's not my project, but I'd rather walk through fire ants. I just can't imagine that it will be easier to change the behavior of the old release to be compatible with what the new release needs than visca versa. Even if the new release has to reshuffle data between pages or stand on its head and wrestle an epiletic porcupine, I still think it will not be as bad as putting the space reservation code in the old release and the new page format in the new release. ...Robert
> for getting 8.4->8.5 upgrade into place. You should first work on an > update process that supports catalog changes, and get that committed. > Once that's in place you'll have enough political capital to prevent > changes in user data layout, and then we'd start to think about schemes > like space reservation that could relax that ground rule. I agree that catalog changes are the best way to start. And I also think it's important that this logic get written in C, primarily so that it gets deeply integrated into core. When a patch submitter adds a function like array_length and puts in the markup for the BKI, they should also be responsible for adding the function to the list of changes that the UIP code needs to make when upgrading from the catversion of the previous release to the current catversion. If upgrade in place consists of Zdenek running around during every beta cycle trying to figure out everything that happened to get done during that dev cycle, it's not going to work (especially if Zdenek gets hit by a bus). That implies a fairly robust and configurable system for adding to and rewriting system catalogs, which today we haven't got. Once we do have it, we can think about how to handle things like page layout bumps and toast chunk changes. Frankly, with things as they are today, there's no way I'm using pg_upgrade on my production system. Even if the whole thing gets rewritten in perl, it's an ex post facto attempt to catch up with a gazillion changes that got made without any regard to whether they'd work with pg_upgrade. It seems very difficult to be confident that this will be 100% correct... ...Robert
Robert Haas <robertmhaas@gmail.com> writes:
> That implies a fairly robust and configurable system for adding to and
> rewriting system catalogs, which today we haven't got.
And we won't ever have, because it's unnecessary and would be impossibly
complex. We know how to do the catalog update: basically, dump, initdb,
reload, then move the data in. There are some corner case issues like
how to preserve toast table OIDs, but the idea that we are going to
invent a special process for each catalog change is just not reasonable.
regards, tom lane
On Tue, 27 Jan 2009, Robert Haas wrote: > That implies a fairly robust and configurable system for adding to and > rewriting system catalogs, which today we haven't got. Right, and designing such a system requires a fairly deep analysis of the catalog changes that have historically shown up, to make sure you've got something that will handle them. The first step here that bears some immediate fruit would be to sit down and analyze everything that changed in the catalog from 8.3 to 8.4. Then you can do two things. You can confirm whether the path used by pg_upgrade really can be expected to work. You could also then talk intelligently about what a design for a catalog rewriting system would look like, using "something that could have handled all of these" as your spec. Then it's onto worrying about page format stuff, which you can already get a handle on by looking at 8.2->8.3 under the same lens. Tom even generated a helpful starting list of things to chase down a few months ago: http://archives.postgresql.org/pgsql-hackers/2008-09/msg00451.php As someone who has been investigating this particular path, the part that troubles me is that you'd have to pushing a new and completely non-glamorous overhead burden ("create an in-place implementation for your catalog change") on people who don't necessarily care about that particular goal, which would create considerable new friction for patch submission that required a catversion bump. Not the sort of boring work volunteers on a project are traditionally good at, and you can expect any developers who aren't managing databases too large to dump/reload to scream about how it will slow the advance of the project. -- * Greg Smith gsmith@gregsmith.com http://www.gregsmith.com Baltimore, MD
Tom Lane wrote: > Robert Haas <robertmhaas@gmail.com> writes: >> That implies a fairly robust and configurable system for adding to and >> rewriting system catalogs, which today we haven't got. > > And we won't ever have, because it's unnecessary and would be impossibly > complex. We know how to do the catalog update: basically, dump, initdb, > reload, then move the data in. There are some corner case issues like > how to preserve toast table OIDs, but the idea that we are going to > invent a special process for each catalog change is just not reasonable. Right, the dump+initdb+reload approach works quite well in both pg_upgrade and pg-migrator. I believe the biggest issue with that ATM is supporting dropped columns, and maybe there's something else, but it's fairly robust and works across any versions. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com
Heikki Linnakangas píše v st 28. 01. 2009 v 09:24 +0200: > Tom Lane wrote: > > Robert Haas <robertmhaas@gmail.com> writes: > >> That implies a fairly robust and configurable system for adding to and > >> rewriting system catalogs, which today we haven't got. > > > > And we won't ever have, because it's unnecessary and would be impossibly > > complex. We know how to do the catalog update: basically, dump, initdb, > > reload, then move the data in. There are some corner case issues like > > how to preserve toast table OIDs, but the idea that we are going to > > invent a special process for each catalog change is just not reasonable. > > Right, the dump+initdb+reload approach works quite well in both > pg_upgrade and pg-migrator. I believe the biggest issue with that ATM is > supporting dropped columns, and maybe there's something else, but it's > fairly robust and works across any versions. It works but it is not ideal. Supporting dropped column requires lot of magic which probably will decrease robustness. When you have a tablespace then there is another shuffling magic which does not seems like something robust. And very important thing is that you need old version of postgreSQL installed, which is something what packagers does not want. Look on Oracle how does it. I have idea how to do it without old PostgreSQL version and with bootstrap process extension which should not be invasive and easily maintainable. I will send idea latter ... stay tuned ;-) Zdenek
* Zdenek Kotala (Zdenek.Kotala@Sun.COM) wrote:
> And very important thing is that you need old version of postgreSQL
> installed, which is something what packagers does not want. Look on
> Oracle how does it.
Just as a counter-point, Debian handles multiple concurrently installed
versions of PostgreSQL just fine, in large part to specifically deal
with the smooth migration challenge (though also because we realize
people may want to continue using the old version while others may want
to install the new version).
Not sure if that's something the community wants to encourage other
packagers to do or if we should look at making it easier to do, but it's
at least possible and has been done for a pretty large distribution.
Thanks,
Stephen
Zdenek Kotala <Zdenek.Kotala@Sun.COM> writes:
> Heikki Linnakangas píše v st 28. 01. 2009 v 09:24 +0200:
>> Right, the dump+initdb+reload approach works quite well in both
>> pg_upgrade and pg-migrator. I believe the biggest issue with that ATM is
>> supporting dropped columns, and maybe there's something else, but it's
>> fairly robust and works across any versions.
> It works but it is not ideal.
Sure, but the other way is just a complete non-starter. It's enormous
amounts of work even for simple changes, and it's not hard to think of
potentially-desirable catalog rearrangements for which it would be
completely unworkable.
> I have idea how to do it without old PostgreSQL version and with
> bootstrap process extension which should not be invasive and easily
> maintainable. I will send idea latter ... stay tuned ;-)
New ideas welcome of course.
regards, tom lane
Stephen Frost <sfrost@snowman.net> writes:
> * Zdenek Kotala (Zdenek.Kotala@Sun.COM) wrote:
>> And very important thing is that you need old version of postgreSQL
>> installed, which is something what packagers does not want. Look on
>> Oracle how does it.
> Just as a counter-point, Debian handles multiple concurrently installed
> versions of PostgreSQL just fine, in large part to specifically deal
> with the smooth migration challenge (though also because we realize
> people may want to continue using the old version while others may want
> to install the new version).
> Not sure if that's something the community wants to encourage other
> packagers to do or if we should look at making it easier to do, but it's
> at least possible and has been done for a pretty large distribution.
The Red Hat/Fedora brigade has also been thinking seriously about that,
though we've not gotten further than thinking yet. Of course, if
pg_upgrade becomes a reality we'd likely stop thinking about it.
IMHO, it would not by any means be a disaster for pg_upgrade to require
a copy of the older-version postmaster. The way I'd foresee packaging
it is to build a separate postgresql-upgrade RPM containing pg_upgrade
itself and a version-N-minus-1 postmaster that gets installed in a
nonstandard location (that pg_upgrade knows about). After you've
finished the upgrade you can remove that RPM and get the extra disk
space back. Most of the possible alternatives mean a *permanent*
disk space hit, because the postmaster will have to contain one-time-use
upgrade code that can't be dropped afterwards.
regards, tom lane
On Wed, Jan 28, 2009 at 10:52 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > Zdenek Kotala <Zdenek.Kotala@Sun.COM> writes: >> Heikki Linnakangas píše v st 28. 01. 2009 v 09:24 +0200: >>> Right, the dump+initdb+reload approach works quite well in both >>> pg_upgrade and pg-migrator. I believe the biggest issue with that ATM is >>> supporting dropped columns, and maybe there's something else, but it's >>> fairly robust and works across any versions. > >> It works but it is not ideal. > > Sure, but the other way is just a complete non-starter. It's enormous > amounts of work even for simple changes, and it's not hard to think of > potentially-desirable catalog rearrangements for which it would be > completely unworkable. Well, the goal of coding is to make such things easier. Already the solution you're advocating has one huge wart: the need to represent dropped columns in pg_dump output. It seems optimistic to assume that there won't be any more, and it seems possible that some change might happen that requires the wart without the necessary wart actually getting inserted. We'll end up testing after the fact to see whether in-place upgrade has gotten broken, and given that no one was under any pressure to give it some forethought in advance, the answer will probably be "yes", at least sometimes... I'm not dead set against doing it this way, I'm just not sold on it. ...Robert
Robert Haas <robertmhaas@gmail.com> writes:
> On Wed, Jan 28, 2009 at 10:52 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> Sure, but the other way is just a complete non-starter.
> Well, the goal of coding is to make such things easier. Already the
> solution you're advocating has one huge wart: the need to represent
> dropped columns in pg_dump output.
Dropped columns are certainly an issue, and TOAST pointers are another,
but they seem to me to be soluble with relatively limited impact
(I don't think pg_dump necessarily needs to be involved; rather I'd
imagine pg_upgrade itself adjusting the new catalog entries after it's
used pg_dump to do most of the work). Trying to do catalog upgrade
in-place is going to be a complete mess. I'd be interested to know,
for example, how you imagine rearranging the contents of pg_class would
work. You don't get to modify pg_class if you can't even find it, which
you can't because you can't read it. And in the time it takes you to
think of an approach for that, I will be able to think of a dozen more
that are all equally nasty. There are probably some thousands of places
in the backend where we expect the system catalogs to have layout
matching what the code expects.
regards, tom lane
Tom Lane píše v st 28. 01. 2009 v 14:06 -0500:
> Trying to do catalog upgrade
> in-place is going to be a complete mess. I'd be interested to know,
> for example, how you imagine rearranging the contents of pg_class would
> work. You don't get to modify pg_class if you can't even find it, which
> you can't because you can't read it.
It is relatively easy. There are very high design:
1) upgrade move old catalog table into save location. We can create for
example special tablespace for it
2) bootstrap creates catalog + create records for old catalog structure
3) data will be copied like
INSERT INTO pg_class select from pg_upgrade.pg_class where oid>16...
In reality it will be more complicated command but pure SQL.
Zdenek
On Wed, Jan 28, 2009 at 5:00 PM, Zdenek Kotala <Zdenek.Kotala@sun.com> wrote: > Tom Lane píše v st 28. 01. 2009 v 14:06 -0500: >> Trying to do catalog upgrade >> in-place is going to be a complete mess. I'd be interested to know, >> for example, how you imagine rearranging the contents of pg_class would >> work. You don't get to modify pg_class if you can't even find it, which >> you can't because you can't read it. > > It is relatively easy. There are very high design: > > 1) upgrade move old catalog table into save location. We can create for > example special tablespace for it > > 2) bootstrap creates catalog + create records for old catalog structure > > 3) data will be copied like > > INSERT INTO pg_class select from pg_upgrade.pg_class where oid>16... > > In reality it will be more complicated command but pure SQL. I really like this idea, assuming I understand it. Basically, I think you're proposing that we move the old system catalogs out of the way, bootstrap a new catalog, and then using SQL (running inside a standalone backend?) to migrate data from the old catalog to the new one. That sounds really good to me. Even a relatively complicated catalog reorganization should be able to be handled this way without too much work or a lot of special-purpose code. ...Robert
Robert Haas <robertmhaas@gmail.com> writes:
> I really like this idea, assuming I understand it. Basically, I think
> you're proposing that we move the old system catalogs out of the way,
> bootstrap a new catalog, and then using SQL (running inside a
> standalone backend?) to migrate data from the old catalog to the new
> one. That sounds really good to me. Even a relatively complicated
> catalog reorganization should be able to be handled this way without
> too much work or a lot of special-purpose code.
Well, what it really means is that all the special-purpose conversion
code is in SQL instead of C. Which is sort of good as long as whatever
transformation you have in mind can be done easily in SQL. (Good luck
if you need to control the OIDs of the inserted rows, for instance.
And I *really* want to see Zdenek handle conversion of stored-rule query
trees in SQL...) But far more importantly, it doesn't fix the problem
that you have to write conversion code in the first place.
The appeal of the pg_dump approach is that it will automatically handle
everything that there exists a plain-SQL representation for, which is to
say darn near everything. We will need special purpose code to deal
with the dropped-column and TOAST-oid issues, but that can probably be
written in SQL if it makes anyone feel better to do so ;-). The more
important point is that once we're done with those two issues, we're
done, and will stay done for the foreseeable future (at least with
respect to catalog upgrades).
I am not sure why everyone is so hot to create a conversion path that
guarantees extra maintenance pain for every future catalog
reorganization, when it would be no more complex to create one without
such a burden.
regards, tom lane
On Wed, Jan 28, 2009 at 6:05 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > Well, what it really means is that all the special-purpose conversion > code is in SQL instead of C. Which is sort of good as long as whatever > transformation you have in mind can be done easily in SQL. (Good luck > if you need to control the OIDs of the inserted rows, for instance. > And I *really* want to see Zdenek handle conversion of stored-rule query > trees in SQL...) But far more importantly, it doesn't fix the problem > that you have to write conversion code in the first place. > > The appeal of the pg_dump approach is that it will automatically handle > everything that there exists a plain-SQL representation for, which is to > say darn near everything. We will need special purpose code to deal > with the dropped-column and TOAST-oid issues, but that can probably be > written in SQL if it makes anyone feel better to do so ;-). The more > important point is that once we're done with those two issues, we're > done, and will stay done for the foreseeable future (at least with > respect to catalog upgrades). > > I am not sure why everyone is so hot to create a conversion path that > guarantees extra maintenance pain for every future catalog > reorganization, when it would be no more complex to create one without > such a burden. Well, I don't personally believe that there will be no extra maintenance pain associated with the pg_dump approach. In fact, the extra maintenance pain will be exactly proportional to the difference between the "darn near everything" that it handles and "everything". Basically, every time you invent a feature that can do things to a system catalog that aren't visible at the SQL-level, you're going to experience the searing pain of having to invent SQL-ish syntax that can be dumped-and-restored without losing that mysterious system catalog magic. The first problem with that is that it is really ugly. Full stop. The second problem with that is that you are relying on your ability to translate tuples in a database table into text format (and not, mind you, the same text format that we normally use to back up and restore databases, but some modified text format with special hacks that are only used when we need to represent things like dropped columns) and then to translate that text format back into a set of new tables that are semantically identical to the original ones in every particular. To my way of thinking, this is a Rube Goldberg machine. Transforming one set of tuples into a slightly different set of tuples using SQL seems way less prone to errors and omissions. I also kind of think that it might open the door to using the system catalogs to indicate things like "the earliest page version that appears in relation X". There's no joy in inventing some kind of pg_dump syntax for that sort of thing just so that you can set it properly when someone does an in-place upgrade. It's useless for normal operation since any NON-uip relations will always have whatever the current version is in that field, and it feels wrong for users to have the ability to fiddle with that value via SQL anyway. With respect to the specific problems you mention, OIDs are definitely an issue but do you think that's an insurmountable obstacle? Seems like we should be able to find a hammer large enough to solve that problem. As for rules, just because the core of the engine is written in SQL doesn't mean that it can't make outcalls to C functions; we already have an interface for that. It is better than writing the whole thing in C, to be sure... I don't know, I'm not completely sure how hard this will be, or which approach is better. But it sounds to me like this has potential, if done right. ...Robert
Tom Lane wrote: > The appeal of the pg_dump approach is that it will automatically handle > everything that there exists a plain-SQL representation for, which is to > say darn near everything. We will need special purpose code to deal > with the dropped-column and TOAST-oid issues, but that can probably be > written in SQL if it makes anyone feel better to do so ;-). The more > important point is that once we're done with those two issues, we're > done, and will stay done for the foreseeable future (at least with > respect to catalog upgrades). > > I am not sure why everyone is so hot to create a conversion path that > guarantees extra maintenance pain for every future catalog > reorganization, when it would be no more complex to create one without > such a burden. I am stumped as well. In the 12 years I have been involved, there are perhaps five issues that the original pg_upgrade written in 1998 didn't handle, and mostly handles now. Considering the number of catalog changes since 1998, the ratio is enormous. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. +
On Thursday 29 January 2009 01:05:07 Tom Lane wrote: > The appeal of the pg_dump approach is that it will automatically handle > everything that there exists a plain-SQL representation for, which is to > say darn near everything. We will need special purpose code to deal > with the dropped-column and TOAST-oid issues, but that can probably be > written in SQL if it makes anyone feel better to do so ;-). Dropped columns are certainly solvable. You just include the dropped column in the dumped CREATE TABLE statement and then issue a DROP COLUMN statement afterwards. You might have to do some extra work if there is a name conflict between a dropped and a later-added column, but that shouldn't be so hard. All you need is the space, not the column names, after all.
Hm the previous proposal was to add syntax to create table to create placeholder columns of specified width. On the one hand the special syntax is less kludgy but on the other hand keeping all the compatibility code in pg_dump is attractive. Net I think prefer your solution. I don't think name conflicts are a problem either since the original name is long gone anyways. All you need to do is find a type with a matching width and make a an arbitrary dummy name. -- Greg On 29 Jan 2009, at 09:33, Peter Eisentraut <peter_e@gmx.net> wrote: > On Thursday 29 January 2009 01:05:07 Tom Lane wrote: >> The appeal of the pg_dump approach is that it will automatically >> handle >> everything that there exists a plain-SQL representation for, which >> is to >> say darn near everything. We will need special purpose code to deal >> with the dropped-column and TOAST-oid issues, but that can probably >> be >> written in SQL if it makes anyone feel better to do so ;-). > > Dropped columns are certainly solvable. You just include the > dropped column > in the dumped CREATE TABLE statement and then issue a DROP COLUMN > statement > afterwards. You might have to do some extra work if there is a name > conflict > between a dropped and a later-added column, but that shouldn't be so > hard. > All you need is the space, not the column names, after all. > > -- > Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) > To make changes to your subscription: > http://www.postgresql.org/mailpref/pgsql-hackers
Bruce Momjian píše v st 28. 01. 2009 v 23:19 -0500:
> I am stumped as well. In the 12 years I have been involved, there are
> perhaps five issues that the original pg_upgrade written in 1998 didn't
> handle, and mostly handles now. Considering the number of catalog
> changes since 1998, the ratio is enormous.
You don't need convert all of catalogs and only few of them are usually
modified in each version (like pg_class, pg_type...).
Zdenek
Tom Lane píše v st 28. 01. 2009 v 18:05 -0500: > Robert Haas <robertmhaas@gmail.com> writes: > > I really like this idea, assuming I understand it. Basically, I think > > you're proposing that we move the old system catalogs out of the way, > > bootstrap a new catalog, and then using SQL (running inside a > > standalone backend?) to migrate data from the old catalog to the new > > one. That sounds really good to me. Even a relatively complicated > > catalog reorganization should be able to be handled this way without > > too much work or a lot of special-purpose code. > > Well, what it really means is that all the special-purpose conversion > code is in SQL instead of C. Which is sort of good as long as whatever > transformation you have in mind can be done easily in SQL. (Good luck > if you need to control the OIDs of the inserted rows, for instance. > And I *really* want to see Zdenek handle conversion of stored-rule query > trees in SQL...) But far more importantly, it doesn't fix the problem > that you have to write conversion code in the first place. OK, it was too optimistic to take only SQL. But all problem are should be fixable with C procedure and storing plain ascii for rewrite rules and reparse it again is not so big problem I guess. > The appeal of the pg_dump approach is that it will automatically handle > everything that there exists a plain-SQL representation for, which is to > say darn near everything. We will need special purpose code to deal > with the dropped-column and TOAST-oid issues, but that can probably be > written in SQL if it makes anyone feel better to do so ;-). The more > important point is that once we're done with those two issues, we're > done, and will stay done for the foreseeable future (at least with > respect to catalog upgrades). One one side I was convinced to select on read conversion, because it is simple and self contained. And now when I try to do catalog upgrade to same way, the another solution is preferred which needs many unsystematic hacks into SQL syntax, pg_dump and so on. > I am not sure why everyone is so hot to create a conversion path that > guarantees extra maintenance pain for every future catalog > reorganization, when it would be no more complex to create one without > such a burden. It is not only about catalog. Currently you have FSM which works fine if it is empty, but when somebody invent any new fork which will needs have complete data, then you need to handle it separately anyway. By main opinion to extend parser and add another complexity into already complex part instead of writing store procedure for each affected catalog which convert its data. Both version will work and I think that maintenance pain will not be so big problem. I think the amount of hot modified catalogs is not big. Advantage of direct conversion are: 1) no pg_dump 2) no old database server 3) no shuffling with files/tablespaces 4) it should be relatively robust and easy to use Thats what I can say now It needs more analysis. Zdenek
Zdenek Kotala wrote: > The current project is not in good shape. Feature freeze is coming and > nothing is committed. Currently there are three patches in the game: > > 1) Space reservation > http://archives.postgresql.org/pgsql-hackers/2008-12/msg00886.php > http://archives.postgresql.org/pgsql-hackers/2009-01/msg02031.php > > This patch is mandatory for page online conversion and MUST TO be part > of postgreSQL 8.4. if not ... then we will be at the beginning next > year. > > I sent updated version today. I think we have decided that any system we come up with now for space reservation will be inadequate, so we will just backpatch any needed space reservation code. Fortunately space expansion is a rare event. And we decided we don't need a pg_class column because a custom table will do just as well in reporting the tables that have been prepared with reserved space. > 2) pg_upgrade.sh > http://archives.postgresql.org/pgsql-hackers/2008-12/msg00248.php > > Pg_upgrade.sh is shell script for catalog conversion. It works for > 8.3->8.4 upgrade. It will be useful while we will not have better > solution. Disadvantage is that it is korn shell script. The idea is to > rewrite it in PERL which is more portable, but I'm not PERL expert and > currently there is no workable solution. > > And what is deadline for it? We can delivery it later with 8.4.1, but > best time for testing is during betas and RC period. Now that pg_migrator is BSD licensed, and already in C, I am going to spend my time trying to improve pg_migrator for 8.4: http://pgfoundry.org/projects/pg-migrator/ -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. +
Bruce Momjian wrote: > Now that pg_migrator is BSD licensed, and already in C, I am going to > spend my time trying to improve pg_migrator for 8.4: > > http://pgfoundry.org/projects/pg-migrator/ What is the plan now? Get pg_upgrade working, get pg_migrator working, ship pg_migrator in core or separately? Is there any essential functionality that we need to get into the server code before release? Should we try to get dropped columns working? It's quite late to be wondering about this, so unless we get a clear and definite plan this week, I say we stop kidding ourselves and drop it.
Peter Eisentraut wrote: > Bruce Momjian wrote: > > Now that pg_migrator is BSD licensed, and already in C, I am going to > > spend my time trying to improve pg_migrator for 8.4: > > > > http://pgfoundry.org/projects/pg-migrator/ > > What is the plan now? Get pg_upgrade working, get pg_migrator working, > ship pg_migrator in core or separately? Is there any essential > functionality that we need to get into the server code before release? > Should we try to get dropped columns working? It's quite late to be > wondering about this, so unless we get a clear and definite plan this > week, I say we stop kidding ourselves and drop it. Oh, a plan? ;-) Basically I am trying to add functionality to the code and clean it up so it is easier to maintain. You can grab the CVS to see my current version: http://pgfoundry.org/scm/?group_id=1000235 Everyone seemed to prefer a migration utility in C, and pg_migrator is in C so I am working on that. It has a BSD license now so we could include it if we wanted to. I am working on pg_migrator TODO items. My current list is: o Makefiles are not yet complete.o need to check crc when we reading the pg_control fileo compare the pg_controls of oldand new serverso fix loaded tables with dropped columns; the dropped column location is not part of pg_dump; pg_attribute.attisdroppedmust be checked and a replacement table created and the column droppedo must call vacuum freezeon system tables before clog is copiedo restore pg_database.datfrozenxid to their original valueso restore pg_class.relfrozenxidto their original values I am not sure about the first two items, but I want to try to address the other ones, though I am not sure how to handle the drop column case. Once I am complete the existing TODOs I will start on testing and getting more feedback on missing features. If folks want to help out, please let me know. I am not aware of any server changes needed for 8.3-8.4 migration. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. +
On Thursday 12 February 2009 05:15:02 Bruce Momjian wrote: > I am working on pg_migrator TODO items. My current list is: [long] Can we clarify now that this is not going to ship with the PostgreSQL 8.4 tarball? > I am not aware of any server changes needed for 8.3-8.4 migration. OK, Zdenek, any concerns, or can we consider this chapter closed?
Peter Eisentraut wrote: > On Thursday 12 February 2009 05:15:02 Bruce Momjian wrote: > > I am working on pg_migrator TODO items. My current list is: > [long] > > Can we clarify now that this is not going to ship with the PostgreSQL 8.4 > tarball? That is not really my decision, but I am not going to advocate its inclusion unless there is some unusual ground-swell of demand and others are going to advocate its inclusion, not me. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. +
Peter Eisentraut píše v čt 12. 02. 2009 v 15:30 +0200:
>
> > I am not aware of any server changes needed for 8.3-8.4 migration.
>
> OK, Zdenek, any concerns, or can we consider this chapter closed?
I think, that it is closed now. Space reservation will be backported if
we need it for 8.4->8.5. And I don't know about any "must" changes in
server for migration from 8.3->8.4.
Zdenek